Code skills boost language agent performance and learning in game environments
Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents
Artificial Intelligence
Summary
Language agents often struggle with tasks that need many small, step-by-step actions. This paper studies how using pre-made code skills helps these agents by letting them call reusable actions instead of making every little decision. The researchers tested this in a complex game called NetHack and found that using these skills improved game progress, cut down computing costs, and sped up learning. The agents could still use basic actions when needed, keeping them flexible. The team also released their skill library and code for others to use.
What this means in practice
- •For game developers: Develop AI players that learn faster and perform better in complex game environments using reusable code skills.
- •For robotics programmers: Incorporate code-based skill libraries to improve efficiency and flexibility of language-driven robotic control in tasks requiring many steps.
Authors
Bartłomiej Cupiał, Jens Tuyls, Maciej Wołczyk, Davide Paglieri, Martin Klissarov, Benjamin Eysenbach, Piotr Miłoś, Karthik R. Narasimhan
Abstract
Language agents struggle to act and learn in environments that require long sequences of low-level actions. Code-based abstractions can make these agents more productive by letting them invoke reusable skills instead of repeatedly selecting individual actions. The code handles recurring local decisions, while the language model decides which skills to use and how to combine them. Yet abstractions are leaky, and situations beyond a skill's capabilities may require a return to primitive actions. Motivated by this tradeoff between productivity and flexibility, we systematically study how code-based action abstraction affects the performance, inference cost, and learning of language agents. We study this in NetHack, a challenging, long-horizon game environment, using CodeHack, our library of code-based skills with natural-language descriptions. We use this library to compare agents restricted to primitives with those using semantic skills alone or in combination with primitives. We evaluate these agents in three settings: zero-shot prompting, supervised fine-tuning, and reinforcement learning. Across a broad zero-shot evaluation on NetHack, we find that compared with primitives, skills nearly triple game progression, while reducing inference cost per episode by 86%. Combining skills with primitives retains much of this benefit while preserving a path back down to low-level actions. Finally, in RL, we find that skill-based agents learn significantly faster than agents acting on primitives, achieving a 7.2x larger average gain in dungeon level over the same training budget. These results show that a supplied skill library can improve performance, efficiency, and learning, while retaining primitives provides flexibility when the library is insufficient. We release CodeHack together with training and evaluation code.