Calliope finds topics in private ai conversations without embeddings
CALLIOPE: Private Insights into AI Use, Without Embeddings
Machine LearningArtificial IntelligenceComputation and LanguageCryptography and Security
Summary
People often want to discover topics in large amounts of text, like conversations with AI, while keeping the data private. Traditional methods use number-based summaries called embeddings but can leak information when privacy is strict. The authors created CALLIOPE, a new approach that uses AI language understanding directly instead of relying on embeddings. This method offers better results, especially when strong privacy protections are required.
What this means in practice
- •For data privacy teams: Organize large sensitive text datasets into topics while maintaining strict privacy guarantees without relying on embeddings.
- •For customer service analytics teams: Analyze and categorize large volumes of private AI chat logs to extract insights without compromising user privacy.
Authors
Edith Cohen, Vadym Doroshenko, Badih Ghazi, Charlie Harrison, Pritish Kamath, Alexander Knop, Ravi Kumar, Ethan Leeman, Pasin Manurangsi, Mikhail Pravilov, Adam Sealfon, Yurii Sushko, Marika Swanberg, Chiyuan Zhang
Abstract
We introduce CALLIOPE, a top-down framework for discovering topic hierarchies in large corpora of unstructured data (e.g., LLM conversations), while supporting differential privacy (DP). Unlike traditional and even recent LLM-augmented approaches that ultimately rely on clustering vector embeddings, our framework leverages LLM capabilities directly for the entire pipeline. Experiments demonstrate that this approach provides better utility over existing (embedding-based) methods in the DP setting, especially in the high privacy regime. We evaluate the quality of CALLIOPE outputs using a variety of traditionally studied clustering metrics, as well as some novel utility metrics that we introduce.