Autonomous research reaches near top results in telecom ticket retrieval
Autonomous Research for Open-Ended Problems: A Case Study on Telecom Ticket Retrieval
Artificial IntelligenceInformation RetrievalMachine Learning
Summary
Solving real-world, open-ended machine learning problems without human help is very challenging. The authors studied how autonomous AI systems perform in telecom ticket retrieval, which involves finding relevant records from customer support tickets. They found these systems do well at tuning specific settings but still lack human creativity and intuition. With some human guidance, the AI got most of the way to the best results much faster and at low cost. The study suggests combining human and AI efforts works best for complex research tasks.
What this means in practice
- •For telecom data engineers: Improve ticket retrieval systems faster using autonomous research with minimal human tuning for near top-tier performance.
- •For customer support platform developers: Use autonomous agents to optimize machine learning models for large-scale text retrieval tasks in customer service workflows.
Authors
Junghyun Min, Huseyin Uzunalioglu, Mohamed Trabelsi
Abstract
Recent breakthroughs in LLM-based systems and their abilities in problem solving and coding have allowed progress in the AI for Science paradigm, potentially replacing human roles in machine learning (ML) research. However, while several frameworks of fully autonomous end-to-end ML research have been proposed, successful implementations of them are often limited to problems with narrow search spaces, like language modeling or biomedical ML benchmarks. In this paper, we explore how autonomous research can be adapted to solve open-ended, industry-grade ML problems, by considering a case study: telecom ticket retrieval, an open-ended task with degrees of freedom in representation, architecture, and training data generation. We discover that autonomous research for open-ended problems with commercial and open-source agents shows both promise and limitations: while autonomous research can excel in narrow hyperparameter optimization, it lacks human-like intuition and creativity and requires operational overhead. Even with minimal human supervision, autonomous research can reach $90\%$ of state-of-the-art performance (0.34 vs. 0.38 Recall@1) in a much shorter time period (10 weeks vs. 10 months of human work) at a modest cost (up to \$200 per Cursor campaign). Our empirical evidence recommends that human researchers and autonomous research frameworks work together for best results in ML research.