Enterprise agents struggle with evolving multimodal workplace records
MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records
Artificial Intelligence
Summary
Workplace information often changes and is shared through documents, chats, and images, making it hard for AI agents to answer questions reliably. The authors created a test called MMEnterprise-Bench with over 1,600 questions to see how well AI handles changing and conflicting information from multiple sources. They found current models have trouble when information conflicts or comes from multiple images, and simply retrieving more data doesn’t always help. This benchmark helps measure progress in building better AI agents for complex workplace environments.
What this means in practice
- •For enterprise software developers: Evaluate and improve AI assistants handling changing project documents, chats, and images over time to support team collaboration accuracy.
- •For legal compliance teams: Test AI tools that verify consistent records and flag conflicting information across multiple document revisions.
Authors
Seohyun Lee, Jongha Kim, Dohwan Ko, Ji Soo Lee, Seoung Choi, Jayun Ku, Hyunwoo J. Kim
Abstract
Enterprise agents answer questions over workplace records that evolve through team collaboration. As information is shared, corrected, and revised across documents, chats, and figures, multiple versions of the same information accumulate in the workspace. Reliable question answering therefore requires agents to reason over sufficient evidence, resolve cross-source and temporal conflicts, and recognize when the available evidence is insufficient to establish an answer. To systematically evaluate these capabilities, we introduce MMEnterprise-Bench, a benchmark comprising 1,613 questions grounded in evolving multimodal records from simulated collaboration projects, covering three categories: Evidence Reasoning, Conflict Resolution, and Unanswerability. We evaluate 10 proprietary and open-source multimodal LLMs and find that even the strongest models struggle when evidence contains cross-source or temporal conflicts or spans multiple images. Moreover, increasing the retrieval budget or employing a larger retriever improves evidence recall but does not consistently translate into higher answer accuracy. These findings highlight the persistent challenges of reasoning over evolving workplace records and establish MMEnterprise-Bench as a testbed for developing more reliable enterprise agents. Code and data are available at https://mlvlab.github.io/MMEnterprise-Bench/.