Summary
Many AI systems are designed to work on tasks that require them to keep watching and acting over several days or weeks, like checking the stock market or news. The authors say that previous tests for these AI agents do not fully capture the challenges of constantly changing real-world environments. They created a tool called ReLiveGym that plays back real news, market, and social media information over simulated weeks, letting AI agents act and learn over a long period. Their experiments show that when and how these agents decide to act is important, and different tasks and AI models may need different designs. They also find that letting agents learn continuously from feedback can help them perform better in these long time tasks.
What this means in practice
- •For market analysts: Use ReLiveGym to test AI agents handling months-long market data with real timing and feedback to improve trading or analysis strategies.
- •For social media managers: Evaluate automated agents on maintaining and adapting social media campaigns over weeks using realistic replayed social media streams.
Authors
Xisen Jin, Jingheng Li, Zhenglun Chen, Junyi Du, Xiang Ren
Abstract
As large language model (LLM) agents become widely adopted, they are increasingly deployed for tasks that require persistent monitoring or recurring actions (e.g., market analysis). These agents are expected to operate unattended for days or weeks, act at the right timing, and adapt to the dynamic environment over time. These challenges are not fully captured in the existing long-horizon agent work, as they often consider a static environment that is not temporally changing. We introduce ReLiveGym, a diagnostic evaluation environment of long-lived tasks in which agents act sparsely over simulated weeks of chronologically replayed real-world news, market, and social-media streams. The tasks span diverse levels of time sensitivity, reasoning intensity, and recurrence. Across eight base language models, we investigate how model choice and harness design affect agent performance on such long-lived tasks. Our results show that how agents determine when to act arises as an important harness-design axis for long-lived tasks; and that the optimal design varies across tasks and sometimes model choices as well. We also evaluate how continuous learning from hindsight feedback affects performance and addresses failure modes observed in these long-lived tasks. These findings indicate model choice, action timing mechanism, and use of feedback as important considerations in the design of long-lived agents. Code: https://github.com/SaharaLabsAI/ReLiveGym