ProMediConv sets benchmark for AI legal dispute mediators
ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation
Computation and Language
Summary
Mediating disputes is important but training skilled mediators is hard and slow. The authors created ProMediConv, a new way to test AI systems that handle mediation, using real legal cases with detailed notes on strategies and behaviors. They also designed a better method for measuring how well these AI agents influence the conversation over time. Their study reveals challenges current AI models face in handling complex, multi-person legal talks. This work offers a useful foundation and standard to improve AI tools for resolving conflicts.
What this means in practice
- •For legal technology developers: Develop AI mediation agents that better handle multi-party legal dispute conversations with validated metrics and real case data.$Commercial implications: Enables creation of legal AI mediation products that improve dispute resolution efficiency and reduce mediator training costs.
- •For customer support teams: Improve multi-party conversation management tools by incorporating mediation strategies and behavior tracking from real dialogues.
Authors
Zesheng Wei, Mengfan Li, Wenhao Liu, Yixin Zhang, Zilei Wang, Yang Deng
Abstract
Dispute mediation is essential for maintaining social harmony and resilience, yet developing skilled mediators is costly and time-consuming. Existing LLM-based mediation research remains limited by unrealistic task formulations, low-fidelity datasets, and coarse evaluation metrics that obscure turn-by-turn dynamics. To address these gaps, we introduce ProMediConv, a novel benchmarking framework that models mediation as a proactive, multi-stage, and party-aware dialogue process incorporating 11 mediation strategies and four party behavior pattern (BP) states. Using 972 complete real-world cases, we construct a high-fidelity mediation dataset with utterance-level annotations of strategies and BP states. Furthermore, to better assess agent impact, we propose MAD (Mean Attribute Difference), a fine-grained metric that captures BP shifts throughout the dialogue. Leveraging this framework, we establish a comprehensive benchmark by evaluating diverse models alongside our tailored baseline ProMediAgent. Extensive empirical analyses reveal critical behavioral phenomena and underscore the persistent challenges current models face in dynamic, multi-party mediation. Ultimately, ProMediConv provides a rigorous foundation and a vital quantitative standard for advancing AI-assisted conflict resolution. Our dataset and codebase are accessible at https://github.com/ZsWei66/ProMediConv_repo.