Benchmarking REST API testing tools detects microservice interaction failures
RESTberus: Trace-Aware Benchmarking of Automated REST API Testing Tools Across Microservice Dependencies
Software Engineering
Summary
Testing tools for REST APIs often check services one at a time and miss problems caused when services talk to each other. The authors created RESTberus, a system that watches not just the API results but also the detailed traces of how requests move through multiple services. This helps it find and separate failures caused inside a service from those caused by other connected services. Their tests show that many tools miss about one in ten failures this way and that RESTberus finds these extra problems reliably.
What this means in practice
- •For microservice developers: Evaluate and compare REST API testing tools more accurately by detecting failures across service dependencies, improving test coverage in complex deployments.
- •For cloud platform operators: Integrate trace-aware benchmarking of API tests on Kubernetes clusters to find issues arising from inter-service communication in production-like environments.
Authors
John Toniutti, Massimiliano Baldo, Marino Miculan, Vincenzo Riccio
Abstract
Existing REST API testing benchmarks focus on isolated services, overlooking failures arising from interactions with downstream dependencies. In microservice systems, response bodies and entry-point operations alone cannot reliably distinguish failures, compromising tool comparisons. We present RESTberus, a Kubernetes-based benchmarking platform for black-box REST API testing tools on deployed microservice systems. By complementing API-level evidence with distributed traces, RESTberus can distinguish failures manifesting in the target service from those manifesting in downstream dependencies. Across six tools on two microservice benchmarks, our trace-aware deduplication strategy achieves 87.1% accuracy, outperforming baselines from prior work by 10-13 percentage points. Our empirical study shows that all tools expose downstream failures, accounting for 11.7% of estimated unique failures on average, which isolated-service benchmarks cannot expose. On single-service benchmarks, RESTberus produces statistically equivalent results to RESTgym, demonstrating that integration-level assessments can preserve compatibility with existing tools and comparability with prior evaluations.