Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition
2026-06-02 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created NovelAPIBench, a new way to test how well language models can use new programming APIs they haven't seen before. Their benchmark automatically finds new APIs, makes coding tasks, and identifies why models might fail. They found that showing usage examples helps models most, and mixing different types of information is better than using just one kind. Also, fine-tuning models helps them learn how to use information but doesn't replace the need to look up external knowledge. This means both looking up info and training the model are important for generating correct code with new APIs.
large language modelsAPI (Application Programming Interface)code generationbenchmarkingfine-tuningretrievaldynamic benchmarkusage examplesparametric adaptationimport-path errors
Authors
Jinnuo Liu, Yue Peng, Jinhan Niu, Hongyi Wen
Abstract
Large language models for code generation often need to use APIs that are absent from their pretraining data. This requires more than recalling a function name: models must coordinate signatures, module paths, input-output contracts, semantics, and executable usage patterns. Existing novel-API benchmarks are typically static, rely on coarse pass/fail metrics, or use synthetic APIs that may not reflect real library evolution. We introduce NovelAPIBench, a fully automated dynamic benchmark that, for any base model and target library, discovers novel APIs, extracts decomposed knowledge bundles, generates executable coding tasks, and assigns failed samples to six diagnostic categories. Across about 1.9K tasks, four base models, and five domains, we compare knowledge injected through retrieval with knowledge internalized through parametric adaptation. We find that knowledge components are not interchangeable: usage examples are the strongest standalone signal, while the best two-component setting pairs signatures with either mechanisms or examples depending on the domain and backbone. Adding more context, especially source code, can hurt by increasing import-path errors. Parametric adaptation also does not replace retrieval once external knowledge is removed; rather, fine-tuning mainly teaches models how to use provided bundles, and this ability transfers to held-out libraries. These results suggest that retrieval and tuning play complementary roles: retrieval supplies volatile API content, while tuning improves procedural integration.