Serving stacks can distort tool-use behavior in AI model evaluation
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
Computation and LanguageArtificial IntelligenceSoftware Engineering
Summary
When testing AI models that call external tools, the way the system handles those calls can change the test results. The authors found that differences in software layers that serve the models can make it look like models perform poorly or well, even if the models themselves behave the same. They show that some failures are caused by the serving environment and not by the model. The authors suggest careful checks to make sure evaluation measures model behavior and not quirks of the serving software.
What this means in practice
- •For machine learning engineers: Improve evaluation setups to distinguish serving system errors from model errors during AI tool-use testing.
- •For ai system integrators: Ensure tool invocation protocols handle serving stack differences to get accurate model performance metrics.
Authors
Lijuan Tang, Yuemeng Zheng
Abstract
A coding agent must emit a valid tool call--a parseable invocation of a tool in the provided schema--before the harness can execute its chosen action. We study how local serving stacks affect this protocol step and show that measured outcomes can depend on the serving layer rather than model behavior alone. In Ollama, the default tools= request is gated per model by a static template flag: some models are accepted and return calls as text, some return native tool_calls, while Phi-3 and Gemma-3 are rejected before inference. In our harness, rejection and retry exhaustion are not preserved as structured failure metadata, so downstream analysis can misclassify them as model non-calls and naively report 0% fidelity. Adding a text tool list while retaining the native channel recovers much of the measured fidelity for accepted models, whereas a uniform text protocol reduces fidelity for Llama-3.2, which has native tool-call support. Cross-stack probes on Ollama, llama.cpp, vLLM, and SGLang show different handling of the same request. Constrained decoding removes parse failures but can induce non-termination, and turn-pooled versus per-instance estimates differ by up to about 55 points. We conclude with a checklist for treating serving behavior as part of the evaluation protocol.