LLM agents improve themselves without step-by-step instructions

Do LLM Agents Still Need to Be Told How to Self-Improve?

Machine LearningArtificial Intelligence

Summary

This paper explores whether advanced language models can figure out how to learn better on their own, without humans telling them exactly what to do. The authors gave a frozen language model tools and a general goal to improve through practice in game playing, letting it experiment freely. They found that capable models managed to create their own strategies for self-learning and got better over time. These improvements also helped in different types of games and challenges, showing that models can guide their own learning process effectively.

What this means in practice

  • For software developers: Build AI agents that autonomously improve their learning strategies using basic tools and general goals without hardcoded instructions.
  • For game designers: Use autonomous LLM agents to generate gameplay strategies and adapt to new game mechanics without explicit programming.

Authors

Shobhita Sundaram, Cansu Sancaktar, Phillip Isola

Abstract

LLM agents increasingly drive their own learning. Self-improvement methods delegate various components of model optimization to the models themselves. As models become more capable, this motivates a basic question: do these increasingly capable models still need us to specify how they should self-improve? In this paper we study an autonomous paradigm in which, given basic primitives, resources, and a general goal for self-improvement, agents construct their own self-learning algorithm for context-based optimization. Using the setting of gameplay as a testbed, a frozen LLM agent is given a compute budget and tools for collecting experience and updating text documents, and allowed to freely experiment without an externally imposed loop or access to real-world data. We find that sufficiently capable models do self-improve under these conditions, demonstrating strategies that align with classic cognitive frameworks for self-regulated learning and constructs from machine learning literature on self-generated curricula. Furthermore, we study the extent to which the learned text artifacts transfer to external benchmarks, observing improvement across different game horizons, mechanics, and 3D navigation.