Cost-aware ticket routing reduces wrong IT support assignments by nearly 40 percent
Cost-Aware Abstention for LLM-Based IT Ticket Classification
Machine LearningArtificial IntelligenceComputation and Language
Summary
Automatic sorting of IT support tickets can sometimes cost more if tickets are routed incorrectly. The authors tested a method that lets the sorting system choose when to automatically decide and when to ask for human help by understanding the costs of mistakes. They found that adjusting when the system abstains from making a decision greatly lowers wrong assignments and reduces overall cost. Their tests compared traditional classifiers and large language models and showed that cost-aware abstention with simple models can perform better in this task.
What this means in practice
- •For it support teams: Implement cost-aware abstention to reduce expensive misrouted support tickets by using thresholds selected on historical data.
- •For customer service system developers: Integrate calibrated abstention policies in ticket routing AI to improve routing accuracy and cost-effectiveness under customized cost models.
Tested on one dataset.
Authors
Sachin Gupta
Abstract
Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under the illustrative cost model (c_ok, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% CI [35.7%, 40.4%]), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat both always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis. The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and correctness AUROC 0.500. We therefore report correctness AUROC and tie-aware risk-coverage area alongside ECE. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19/1,000. An LLM-only agreement control and margin/entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs; the work is an evaluation framework, not a new abstention algorithm or universal deployment certification.