Evaluating usefulness of AI generated taxonomies on customer feedback
From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback
Artificial IntelligenceMachine LearningSymbolic Computation
Summary
When AI organizes large amounts of customer feedback into categories, the way the categories are arranged affects how well problems are understood and assigned to teams. The authors study AI-generated category trees that look good by usual checks but often repeat category names and overlap in confusing ways. They propose new tests to see if these trees truly split feedback into useful groups for teams to handle. Their work shows that just checking if a tree looks plausible is not enough to know if it will work well in practice.
What this means in practice
- •For customer experience teams: Measure and improve how customer feedback is grouped to ensure each team owns distinct, relevant problem areas.
- •For data engineers: Use whole-tree metrics to validate AI-generated taxonomies before deploying them for routing customer support issues.
Authors
Prabhath Chellingi, Raviraja G, Viraj Bagal
Abstract
Taxonomies are the symbolic representations through which AI systems organize evidence, aggregate patterns, and answer questions over large document collections. Over customer feedback, the category tree decides how every record is counted and routed, which problems get seen, and which team owns them. Agentic harnesses now make it easy to generate a plausible-looking hierarchy, and such trees are checked today with generic, individually scoped checks: each name fits its description, sits under the right parent, and stays distinct from its siblings. We ask a more operational question: when is a generated hierarchy actually useful as a production taxonomy? We build six taxonomies over two proprietary feedback corpora (1,940 and 5,000 records): for each corpus, a production reference and two repeated runs of the same harness under identical inputs. All six pass every generic naming and structure check, and a deeper product-coverage check even prefers the generated trees. Yet in every generated tree at least 97.7% of leaf names merely restate an ancestor's name (13.9% and 2.9% in the references), and in one, three of every four records fall under multiple top-level categories. We introduce two families of whole-tree metrics: structural discriminators test whether a tree's shape was learned from the data or imposed by its generator; team partitionability tests whether branches split feedback into groups teams can own. Trees the generic checks rate as equally correct differ by 27 percentage points in cross-branch leakage, and only one of the two beats a random split. Surface plausibility is an insufficient measure of taxonomy quality: evaluation must measure the whole tree as well as each node.