Users often ask AI for personalized legal help during conversations
Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations
Computation and LanguageComputers and Society
Summary
People sometimes ask chatbots questions about their legal rights and problems. The authors studied how often and in what ways people do this by looking at over 2,000 real conversations. They created categories to identify when users want actual legal advice and what topics they ask about. They also tested computer models to see how well they can recognize these legal questions. Their work helps build better AI systems that understand and respond to legal questions more accurately.
What this means in practice
- •For legal ai developers: Improve AI systems to better detect and respond to user questions seeking personalized legal guidance.
- •For customer support teams: Use legal query detection to route user questions about legal issues to specialized agents or resources.
Authors
Amirreza Mohseni, Gerasimos Spanakis
Abstract
People ask conversational AI models about their legal rights, obligations, procedures, and personal situations that may have legal consequences. To study this behavior, we first need to distinguish requests for legal guidance from other legally themed discussions and unrelated conversations. We contribute a multi-axis taxonomy of legal queries capturing whether legal guidance is sought, how a request is formulated, and which legal domains it concerns; apply it to 2,122 naturally occurring WildChat-1M conversations; and create a human-reviewed gold evaluation set of 200 conversations. All annotations, model checkpoints, and evaluation code are publicly available. In this enriched sample, 71.1% of conversations containing legal content seek personalized or practical guidance. Rights-and-options advice is the most common primary request type among legal-content conversations, and 25.3% of guidance-positive conversations span multiple legal topics. The gold evaluation set supports a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF cascade. Our framework supports large-scale studies of legal guidance in conversational AI, and the public release of the annotated data facilitates reproducible research and future model development.