Automated pipeline creates realistic email sets for enterprise question answering
WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation
Artificial Intelligence
Summary
Answering questions at large companies is hard because information is scattered and often unclear, spread across emails, chats, and documents. The authors built a system that automatically creates synthetic but realistic email data and related questions as if from real workplace projects involving many employees over months. They tested existing question-answering methods on this data and found plenty of room for improvement, showing these tasks remain challenging. This work helps create better benchmarks closer to real enterprise complexity.
What this means in practice
- •For enterprise software developers: Use the synthetic email datasets to test and improve question-answering systems for workplace information retrieval.
- •For customer support teams: Train assistants with complex simulated interactions reflecting real-world ambiguities in enterprise communication.
Authors
Amey Varhade, Ananya Sutradhar, Ravishankar Krishnaswamy, Navin Goyal
Abstract
Enterprise settings provide a challenging environment for question-answering agents, which often rely on Retrieval-Augmented Generation, Deep Research (DR), and related techniques. Much of this challenge comes from the complexity of enterprise data: information is often spread across evolving and potentially conflict- ing emails, chat messages, documents, and other artifacts. Existing benchmarks typically have limited real-world complexity, short-form responses, and unnatural queries, so they often fail to capture the challenges of enterprise settings. In this work, we introduce an automated pipeline for generating synthetic datasets of emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data. Our method simulates long-running enterprise projects spanning several months and involving up to 25 interacting employees across multiple roles. The data emphasizes ambiguity, distributed information, and naturally occurring queries. To validate the pipeline, we evaluate few standard agentic baselines on our datasets using the latest frontier models. We find that aggregate scores averaged over all queries remain below 80% for each dataset, indicating significant room for improvement. These findings suggest that more work remains to be done for enterprise deployment and underscore the importance of realistic, high-complexity evaluation data for developing stronger real-world enterprise DR systems.