Models learn to understand code apis through interactive exploration
Toward Interactive Understanding of Code APIs
Artificial Intelligence
Summary
Understanding how pieces of computer code work can be hard if you only have limited access to them. The authors created a test called PAU that challenges computer models to learn what external code snippets do by asking questions and looking at the answers, rather than reading the code. They found that even the best current models struggle with this task because they often stop exploring too soon. To improve this, they trained models using a technique inspired by robot learning that encourages more active experimentation, leading to better understanding of these code snippets.
What this means in practice
- •For software engineers: Improve debugging and code comprehension tools that learn the behavior of unfamiliar APIs through automated interactive queries.
- •For automation developers: Develop intelligent agents that interactively understand third-party code tools without needing source code access.
Authors
Dhananjay Ashok, Jesse Thomason, Jonathan May
Abstract
Empowered by advances in Language Model agents, systems have made substantial strides in code generation and understanding. However, these approaches often rely on read access to the relevant code, an assumption which does not hold when dealing with external APIs. In this work, we introduce the PAU (Python API Understanding) benchmark, where we provide models with black-box, API-level access to code snippets. Models must query the API with exploratory inputs and draw insights from the resulting outputs, with the goal of describing the snippet's true functionality. By treating the code snippets as external tools that must be understood via interaction alone, PAU studies the more general problem of unsupervised tool understanding, specifically for tools implemented as Python methods. Despite recent progress in coding agents, even frontier models struggle to achieve high performance on PAU, with the best model (Claude-4-Opus) failing to understand over 45% of the PAU test set. An investigation into the common error modes reveals that models are overconfident; they often overrate the quality of their current hypothesis, leading to insufficient exploration and premature termination. Finally, we take inspiration from the Asymmetric Actor Critic (AAC) paradigm, frequently used in robot learning, to post-train models for interactive code understanding. Models trained with AAC conduct more active exploration of the APIs, with an AAC-tuned Qwen3-8B model matching the performance of GPT-5-mini.