Direct Diversity Optimization improves successful AI strategies coverage
Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Many AI agents learn from examples of their past successes, but often they miss out on keeping multiple good ways to solve a task. The authors look at this problem and create a method called Direct Diversity Optimization (DDO), which helps AI keep track of different successful strategies starting from the same point. Their method uses special techniques to group similar decision points and guides the AI to recognize diverse successful options. This approach helps AI perform better and maintain a wider variety of successful strategies in different tasks.
What this means in practice
- •For ai product developers: Improve AI agents’ ability to cover multiple successful strategies in complex tasks, enhancing reliability under limited trial budgets.
- •For robotics software engineers: Build robot controllers that maintain diverse successful action plans from the same decision points to better handle uncertain environments.
Authors
Junwon Ko, Dong-Jae Lee, Minchan Kwon, Sunghyun Baek, Junmo Kim
Abstract
LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget. We present Direct Diversity Optimization (DDO), an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO). DTC constructs state-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference-relative targets over successful alternatives. DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful-only imitation and decoding-time diversification controls.