Attention pooling improves noisy conversational intent classification accuracy

The Hidden States Cookbook: A Large-Scale Ablation Study for Noise-Robust Conversational Intent Classification in Industry

Computation and Language

Summary

Conversational systems often struggle to understand user intentions because people mix in greetings, off-topic remarks, or polite words that confuse machines. The authors tested different ways to combine pieces of a conversation to figure out the main intent better, especially when there is noisy or irrelevant chatter. They found that using attention pooling gave the best results in noisy conversations, while averaging (mean pooling) made performance worse. Filtering techniques in the frequency domain didn’t help much. This work tells developers which methods to use or avoid when building chat systems that work well even with natural, noisy speech.

What this means in practice

  • For chatbot developers: Improve chatbot intent recognition accuracy in noisy user interactions by implementing attention pooling instead of mean pooling.
  • For call center automation teams: Enhance conversational intent classification for customer service systems facing natural, off-topic, or polite speech patterns.

Authors

Bogdan Bogachov, Nikita Letov, Yaoyao Fiona Zhao

Abstract

Conversational database interfaces face a critical challenge: users naturally embed queries in conversational noise (greetings, politeness, off-topic remarks), which degrades intent classification accuracy and wastes computational resources. Despite advances in orchestration and retrieval strategies, a fundamental question remains unanswered: which pooling strategy maximizes intent classification accuracy under realistic conversational noise in production language models? This work addresses this gap through 360 controlled experiments spanning four pooling configurations (mean, max, last-token, attention, and FFT-augmented variants) using Llama-3.2-1B-Instruct on BANKING77 and CLINC150 datasets under clean/noisy conditions with ten random seeds. Key findings reveal that attention pooling consistently outperforms alternative strategies under noisy conditions (~+2.6-2.8 F1 over the default), while mean pooling degrades performance by up to ~5 F1 points. Frequency-domain filtering does not produce consistent accuracy improvements and functions primarily as a structural variation rather than an accuracy-enhancing component. These results provide concrete, evidence-based guidance for building noise-robust conversational classifiers: attention pooling is recommended for noisy interfaces, mean pooling should be avoided, and last-token pooling is appropriate for clean-query scenarios.
View PDFOpen arXiv