Automakers balance cost and control in ai testing tool choices
Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing
Software EngineeringArtificial Intelligence
Summary
Car makers need to check that the touchscreens in their vehicles work well. Volvo Group compared two types of AI tools: custom-made ones they built themselves, and general tools anyone can use. They found that using ready-made tools can save money and sometimes work better, even if they cost more to run. The researchers also found that switching to general tools changes how teams maintain the testing, with less control but less work needed. Testers still want to see what the AI is doing to make sure it follows the correct steps.
What this means in practice
- •For automotive software developers: Reduce overall testing costs by choosing between custom or general AI testing tools for vehicle infotainment systems.
- •For mobile app testers: Improve recovery from incorrect actions during automated tests by using general-purpose AI agent harnesses.
Authors
Arsham Gholamzadeh Khoee, Ruyu Fu, Jiaqi Wei, Ruikun Deng, Robert Feldt, Dhasarathy Parthasarathy, Yinan Yu
Abstract
As general-purpose LLM agent harnesses mature, organizations must decide whether to continue developing in-house systems or adopt existing alternatives. We study this decision at Volvo Group for GUI testing of Android-based in-vehicle infotainment (IVI) systems. We compare a production, custom-built harness with two general-purpose harnesses, Claude Code and OpenCode, keeping the tools, accessed through the Model Context Protocol (MCP), and the models the same. We evaluate the harnesses with up to four models on Volvo's IVI test cases and on open-ended tasks in public apps (Google Maps, Booking.com), and interview ten practitioners: developers, the product owner, and testers. On Volvo's stepwise IVI tests, general-purpose harnesses match the custom-built harness in success but vary more across runs. Their inference cost per task is 2.6-10.7 times higher, yet the engineering effort they save outweighs this: for a nightly IVI regression suite, total monthly cost, including engineering effort and inference, is 3-66% lower. On open-ended public-app tasks, they succeed more often, mainly because they recover better from incorrect actions. Interviews suggest that external advances can turn an in-house harness into technical debt even without deterioration. Adopting a general-purpose harness shifts maintenance from the agent loop to tools and instructions, but gives teams less control over what the model receives. Regardless of the harness, testers want to inspect the agent's actions and ensure that it follows the prescribed test scenario.