Efficiency benchmark measures AI ridehailing agents performance
RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents
Artificial Intelligence
Summary
AI agents that help with ridehailing tasks often get judged just by whether they finish the job. But finishing a task isn't enough if the agent annoys users by asking too many questions or repeating steps. The authors created RideWay, a new test that also checks how efficiently agents work, counting things like extra questions or unnecessary tool uses. They found people care more about the number of extra dialog turns than extra searches, and their metric helps measure this efficiency alongside success.
What this means in practice
- •For ai developers: Compare and improve AI agents handling ridehailing tasks for faster and less repetitive user interactions using a new efficiency-focused benchmark.
- •For customer service teams: Evaluate automated assistants to reduce dialogue friction with users by balancing task success and interaction efficiency in service workflows.
Authors
Qingnuan Han, Boli Fang, Mingzhi Hou, Claire Liu
Abstract
AI agents are usually evaluated by whether they complete a task. In interactive service settings, a successful agent can still frustrate users by asking repeated questions, performing redundant searches, or making avoidable revisions. We introduce RideWay, an efficiency-centered benchmark for ridehailing agents in a stateful tool-calling environment, together with Efficiency Utility, a success-gated metric that discounts successful trajectories for excess tool calls and user-facing turns relative to task-specific reference effort. Human paired preferences calibrate the relative penalties, reflecting an aggregate service-workflow trade-off: extra dialogue often creates visible friction, whereas extra tool use can sometimes verify constraints or preserve user intent. Across 58 tasks and 24 models, the fitted penalty for excess turns is about twice that for excess tool calls. On task-disjoint held-out preferences, Efficiency Utility achieves 78.7% accuracy overall: 90.6% when trajectories differ in turns, but chance-level accuracy when they differ solely in tool calls - the axis on which human annotators agree least. RideWay therefore makes interaction efficiency measurable alongside task success, while exposing the boundary of count-based tool-use evaluation.