Large language models measure effort to use command line tools
Beyond Task Completion: Measuring Interaction Cost in Terminal User Interfaces
Software Engineering
Summary
Using command line tools can be tricky, and just knowing if you finish a task doesn’t tell you how hard it was. The authors created a method where a language model acts like a user to measure how much effort is needed to understand and operate these tools. They tested this on many real tasks and compared human and model efforts, finding differences not shown by simple success rates. This method helps find parts of tools that work but still need to be easier to use.
What this means in practice
- •For software developers: Assess and improve command line tools by measuring user effort beyond task success.
- •For user experience teams: Evaluate terminal interfaces more reliably with automated agents to find usability issues not visible in success rates.
Authors
Ruida Hu, Yuanhao Wang, Chao Peng, Yakun Zhang, Cuiyun Gao
Abstract
Large language models (LLMs) are increasingly used through terminal user interfaces (TUIs), yet task completion alone does not capture how difficult an interface is to understand and operate. Existing human assessments and LLM-generated ratings or reports do not provide repeatable measurements of interaction effort grounded in verified task execution. We propose Agent-as-a-User, an evaluation paradigm that places an LLM agent in the user role. Agent-KLM separates agent-side interpretation and operation costs and relates them structurally to human interaction. TUINaut operationalizes the paradigm by recording interaction trajectories and verifying outcomes with actor-independent oracles. Using TUINaut-Bench, we study 84 tasks across 15 real-world TUIs with six human participants and five observation-evaluator configurations, and evaluate 45 TUIs generated by three leading stacks. Similar aggregate success rates mask differences in which tasks humans and agents complete, whereas interpretation and operation costs follow correlated task rankings, especially for operation. This pattern persists across configurations. Generated TUIs can implement correct functionality while still requiring usability improvements. These results position Agent-as-a-User as a repeatable, execution-grounded paradigm for measuring task-based usability.