Reward rate policy gradient improves efficiency in machine learning agents
Reward-rate Policy Gradient for Efficient Machine Learning Engineering Agents
Machine LearningArtificial Intelligence
Summary
Many machine learning agents take different amounts of time to do tasks like loading data or training models, but traditional methods assume every step takes the same time. The authors proposed a new approach that focuses on the reward earned per unit of time, making the agent more efficient. Their method estimates how well actions perform considering the time they take and adjusts the agent’s decisions accordingly. They tested it on small language models and found it performs better within fixed time limits compared to usual methods.
What this means in practice
- •For machine learning engineering teams: Improve time efficiency of automated model training and feature engineering workflows by optimizing reward rate per action duration.
- •For ai platform developers: Design reinforcement learning agents that better manage variable-duration tasks to deliver higher performance under fixed time budgets.
Authors
Muhang Tian, Sherry Yang
Abstract
Traditional reinforcement learning (RL) techniques focus on maximizing expected cumulative reward, where each action assumes to take a constant unit of time. However, this assumption does not hold for agentic RL tasks such as machine learning engineering (MLE) agents, where actions involve data loading, feature engineering, and model training that take variable durations. Efficiency matters in modern agentic RL where actions are costly. To address this limitation, we adapt from continuous-time RL and Semi-Markov Decision Process (SMDP) formulation and propose Reward-rate Policy Gradient (RPG), where we focus on optimizing the reward rate -- the long-term reward per unit of time. RPG estimates the reward rate from off-policy samples, then charges each action for the time it consumes at that rate. We first conduct theoretical analysis in the bandit setting to establish that RPG approximates the optimal reward rate and empirically demonstrate it outperforms baselines while avoiding enumeration over the policy space, a known issue for an existing method. We then further apply RPG on a small language model (Qwen3.5-4B) with self-improvement loops and empirically show it obtains higher rewards within a fixed time budget than vanilla RL on MLE-Bench and NanoGPT, with a 19.2% and 85.7% margin, respectively. Our method provides a practical solution for optimizing performance under wait time considerations in modern agentic RL tasks, where actions interact with external environments and cost time.