RewardWeaver improves long task learning with adaptive rewards
RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation
Artificial Intelligence
Summary
Learning to complete long tasks is hard because feedback often only comes at the end, making it tough to know which actions helped or hurt. The authors introduce RewardWeaver, a method that changes how rewards are given based on what the AI struggles with during training. By analyzing failures and adjusting rewards to focus on key weak spots, the AI learns more effectively over time. This approach works well on tasks like social chatting, bargaining, and sales, improving the AI's ability to complete complex interactions.
What this means in practice
- •For dialogue system developers: Train conversational AI to better handle long, complex interactions by focusing learning on key failure points dynamically.
- •For automated negotiation teams: Develop AI agents that improve bargaining strategies more efficiently by adapting learning rewards to current weaknesses.
Authors
Hengbo Xiao, Boyao Zhang, Purui Liu, Yuxuan Zheng, Haoran Yin, Haibo Liu, Fan Zhang
Abstract
Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in domains where task outcomes can be reliably evaluated, but long-horizon interaction remains challenging due to sparse terminal feedback and difficult credit assignment. Process rewards provide denser supervision, yet the capabilities most relevant for training can change as the policy evolves: a behavior that is easy to evaluate or frequently deficient need not be the bottleneck currently limiting task success. We introduce RewardWeaver, a self-evolving reward adaptation framework for language agents in long-horizon interaction. RewardWeaver maintains a validated capability space in which the semantics of admitted Rubrics remain fixed, and closes the loop between policy optimization, task evaluation, failure attribution, and reward adaptation. After each training stage, it performs outcome-grounded backward attribution on low-outcome trajectories, aggregates recurrent and policy-controlled capability bottlenecks, and dynamically selects the corresponding process rewards for the next stage. Recurrent failures not covered by the existing capability space trigger a separate, controlled expansion procedure. We evaluate REWARDWEAVER on SOTOPIA, Amazon?HistoryPrice, and a newly constructed Sales Benchmark. Across social interaction, bilateral bargaining, and domain-specific sales, REWARDWEAVER establishes new state-of-the-art (SOTA) results. Ablations further demonstrate the importance of dynamic reward allocation, failure-grounded attribution, and stable semantics for admitted capabilities.