MindEval2 offers new tools to better assess AI mental health chatbots
MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support
Computation and Language
Summary
More people are using AI chatbots for mental health support, but it’s hard to know if the chatbots really help. The authors created MindEval2, which includes detailed checklists made with psychologists that clearly measure the quality of chatbot answers, and a way to simulate more realistic patient conversations over time. Testing 18 different AI models showed that while many have improved on basic aspects, they still struggle with giving proper guidance. This framework helps identify where chatbots need to get better to safely support mental health.
What this means in practice
- •For mental health chatbot developers: Use MindEval2 to test and improve AI chatbots’ ability to provide safe and effective mental health support over time.
- •For healthcare compliance teams: Apply MindEval2’s explicit quality rubrics to evaluate chatbot responses against clinical standards and reduce risk.
Authors
John Mendonça, Daniel Tschernutter, José Pombal, Areti Vassilopoulos, Catarina Botelho, Ignacio Sarasua, Maya D'Eon, Nuno Guerreiro, Ricardo Rei
Abstract
With people increasingly turning to AI chatbots for mental health support, the need to evaluate these tools is becoming more pressing. Recent benchmarks attempt to mimic a realistic setting by simulating multi-turn interactions with synthetic patients and evaluating them with an LLM-as-a-judge. However, the usefulness of these methods is limited because simulated patients can be overly forthcoming and self-reflective, and judgments typically involve opaque Likert scales with unclear practical utility. To bridge these gaps, we introduce MindEval2, an evaluation framework that consists of two components: Mental Health Quality Rubrics (MHQR), for evaluation, and MindSim, for patient simulation. For MHQR, we collaborate with PhD-level psychologists to develop a set of rubrics that break down response quality into explicit operational criteria that are more interpretable and actionable than existing evaluation taxonomies. MindSim, in turn, is an LLM harness that combines carefully-designed patient profiles with tools, enabling more varied, challenging, and realistic interactions than existing baselines. The harness also contains a memory component that is updated between each interaction, enabling MindEval2 to model the longitudinal aspect of mental health. We evaluate 18 proprietary and open-weight models on MindEval2, finding that while frontier models have largely improved on issues related to surface level quality and boundaries, guidance calibration remains the dominant failure mode across all models, accounting for most of the quality gap.