VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
2026-08-12 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created VAKRA, a new test to check how well AI agents can use APIs and information together in complicated ways. It includes over 8,000 real-world APIs across many areas and tests different levels of difficulty, like handling multiple steps and following rules about using tools. They tested top AI models and found that while models do okay on simple tasks, their performance drops a lot when tasks get harder or have rules. Most mistakes happen when the AI has to understand or link information, not when it actually calls the APIs.
APIMulti-hop reasoningKnowledge retrievalAgent architectureReAct frameworkTool-use policiesEntity disambiguationCross-source groundingExecutable APIsBenchmarking
Authors
Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor
Abstract
Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4\% on single-hop endpoint-style tasks and drops to 50--51\% on compositional APIs; performance degrades by over 50\% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4\% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA