BrandEval benchmarks long-term crisis communication for brands
BrandEval: A Two-Track Multi-Agent Benchmark for Risk-Sensitive LLM Crisis Communication
Artificial Intelligence
Summary
Handling product crises well is tricky because responses can affect public trust over time and competitors may react too. The authors introduce BrandEval, a set of tools to test how well AI models handle these situations both in single replies and over long conversations with other agents involved. They created a dynamic sandbox to simulate public feedback and competitor actions to see how communication strategies perform. Their tests show that judging models only on short-term replies misses risks visible over longer interactions. BrandEval helps reveal when AI policies might fail in complex crisis scenarios.
What this means in practice
- •For public relations teams: Evaluate AI communication strategies to manage brand reputation during product crises considering competitor and consumer reactions over time.
- •For chatbot developers: Test and improve crisis response policies for conversational AI in simulated multi-agent environments that mimic real-world brand communication challenges.
Authors
Haotian Gan, Weidong Tang, Hanbin Sun, Yi Luo, Xintian Li, Jiayong Wang, WanYue Li, Yudong Li
Abstract
Evaluating LLMs for crisis communication requires more than judging isolated responses, because public feedback and competitor actions can reshape trust over time. Existing benchmarks remain largely static and single-turn, offering limited evidence about long-horizon risk-sensitive communication. We introduce \textbf{BrandEval}, a two-track benchmark for product-crisis communication, focusing on public brand responses to product-related safety, quality, service, and trust incidents. BrandEval-Static evaluates stakeholder-aware strategic analysis, while BrandEval-Dynamic stress-tests response policies in \textbf{BrandPolis}, a controlled multi-agent sandbox with consumer feedback, competitor responses, and trust-risk trajectories. We also construct \textbf{BrandSRD}, a Chinese crisis-response dataset, and instantiate a reference Strategic Agent using \textbf{Strategic Rationale (SR)}. Across proprietary and open-source models, we find that static strategic quality is informative but incomplete: it aligns with some end-state outcomes but does not reliably predict volatility and delayed recovery; dynamic stress tests further expose tail-risk failures under manipulative or weakly grounded policies. The code and dataset will be available at https://github.com/htgan-ai/BrandEval.