Rised guides training of generalist ai agents across multiple environments

RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation

Artificial Intelligence

Summary

Training AI agents that can work well in many different environments is tricky because feedback is often simple and limited. The authors created a system called RISED that uses detailed textual feedback, called rubrics, to better understand how the agent behaves in different settings. This richer information helps select the most useful training examples and improves the agent's learning by showing both good and bad behaviors. As a result, RISED trains more effective generalist AI agents that perform well across a variety of tasks.

What this means in practice

  • For ai research teams: Train more capable generalist agents by improving environment data selection and supervision with rubric-based feedback.
  • For game ai developers: Improve training of AI agents that must learn to perform well across multiple game environments simultaneously.

Authors

Jingtan Wang, Sirajul Salekin, Young mok Jung, Javier Movellan, Bryan Kian Hsiang Low, Manjot Bilkhu

Abstract

Training a single LLM agent jointly across diverse interactive environments has attracted increasing attention as a route to generalist agents. Existing curriculum and data-selection strategies often allocate training at the environment level or prioritize local reward-based signals, without explicitly considering relationships between current rollouts across environments for prompt-group selection. Meanwhile, as environments are learned at different rates, all-failure and all-success rollout groups can coexist within a batch, leaving those data without group-relative reward signals. Both challenges highlight limitations of relying solely on scalar rewards in multi-environment RL: they provide limited information about cross-environment relationships and no within-group reward contrast when rewards are identical. This motivates richer textual feedback, such as rubrics describing rollout behaviours, to guide learning. Beyond rubrics' usage as reward, we repurpose rubrics to guide both online data selection and policy supervision. An LLM judge tags each rollout using a predefined rubric vocabulary shared across environments. The resulting profiles guide the selection of data that aligns with the overall behavioural composition of the mixed-environment batch while limiting overlap with already-selected data. Available positive rubrics (describing desired behaviours) provide privileged context for an on-policy self-distillation teacher, supplying additional token-level supervision, while negative rubrics (describing undesired behaviours) guide subsequent rollout generation away from recurring failure modes. Together, these components form RISED. Across model backbones, RISED achieves the highest mean pass rate across environments and ranks first or second in every individual environment. Rubric-based analysis of RISED can further characterize the behavioural changes accompanying these gains.