Adaptive circuits improve accuracy and cost of llm classifiers
Circuits of LLM Classifiers
Machine Learning
Summary
Classifying something with language models can be unreliable and inconsistent. The authors see each language model’s guess as a noisy measurement and build a process—called a circuit—that asks questions adaptively to improve accuracy while considering the cost of queries. They analyze how to optimize these circuits and test them on various tasks like health and safety classification. Their approach achieves better accuracy for the amount of effort spent compared to simpler methods.
What this means in practice
- •For ai platform engineers: Design classification services that adaptively combine multiple LLM outputs to maximize accuracy per query cost.
- •For content moderation teams: Use adaptive query circuits to improve detection reliability on sensitive or diverse datasets within constrained computational budgets.
Authors
Hadi Khalaf, Sajani Vithana, Kunal Talreja, Haewon Jeong, Viveck R. Cadambe, Flavio Calmon
Abstract
Language models make it easy to build classifiers by simply prompting a model with the sample to classify. However, their predictions are neither deterministic nor uniformly reliable across samples. How can we combine predictions from different models to improve reliability? We view LLM classifiers as noisy measurement instruments of an underlying target label. Our goal is to maximize average classification accuracy at a given average query cost. To this end, we introduce circuits that organize LLM classifiers into an adaptive sequence of queries and decisions. We then characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit. Across seven classification tasks spanning safety, health, political science, human preference, agent evaluation, decision-making and reasoning, our method improves the cost-accuracy frontier over routing and cascading baselines.