Taxonomy grounding improves AI repeatability in customer feedback analysis
Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis
Artificial Intelligence
Summary
AI tools that organize customer feedback need to give consistent answers when the input data doesn’t change. The paper measures how much these AI outputs vary across repeated runs, even if the answers seem reasonable. The authors find that using a taxonomy-guided approach, which organizes feedback into stable categories and subcategories, greatly reduces these differences. This method works better than others across different models and data sizes. It also applies to other large collections of unstructured text beyond customer feedback.
What this means in practice
- •For customer experience teams: Generate stable and consistent categories from customer feedback to track changes accurately over time without confusing AI categorization shifts.
- •For financial data analysts: Consistently summarize large sets of financial documents to reduce variation in categorization and counts between repeated AI runs.
Authors
Viraj Bagal, Raviraja Ganta, Prabhath Chellingi
Abstract
AI agents are increasingly being programmed to automate knowledge work over large collections of unstructured data. Such automation requires repeatability: when the underlying evidence is unchanged, the agent's categories, priorities, and counts should not shift materially between runs, even if each individual answer appears plausible. We introduce a repeat-run evaluation framework that aligns semantically equivalent categories and focuses on two operating metrics: theme churn, the normalized change in the returned category set, and volume disagreement, the change in counts for categories that persist. We evaluate three recurring customer-feedback tasks across eight frontier models, corpus sizes from 100 to 5,000 records, multiple prompts, and three execution designs: raw generation, taxonomy-free hierarchical decomposition, and a taxonomy-grounded agent (TGA) using persistent themes, subthemes, and record-level predictions. With Claude Opus 4.8 and the 1,000-record corpus fixed, TGA reduces theme churn by 86--88% relative to both raw generation and hierarchical decomposition, while matched-theme volumes have zero disagreement. The taxonomy-grounded agent is more stable than every raw model in the screen, remains more stable at each corpus size, and keeps this advantage when theme matching is made stricter or looser. Although evaluated on customer feedback, the framework targets repeated synthesis of unstructured corpora more broadly, including financial reports, legal documents, incident records, and scientific literature. Overall, these results show that taxonomy grounding produces more consistent and repeatable outputs for recurring knowledge work.