Cross-language legal question answering struggles with retrieval and correction
Cross-Lingual Legal QA for Vietnamese Labour Law: Retrieval, Translation, and Verifier-Guided Correction
Computation and Language
Summary
Answering legal questions across different languages is tricky because it involves finding the right law texts and making sure answers are accurate. The authors focus on Vietnamese labor law and compare different methods to find and check legal answers in English and Vietnamese. They try a method that breaks down answers, verifies if citations are correct, and attempts to fix errors. They found dense search worked better than sparse search in finding relevant laws, but correction methods only slightly improved citation accuracy. Also, automatic checks do not always match human opinions about answer quality.
What this means in practice
- •For legal technology developers: Improve cross-language legal QA systems by integrating verifier-guided pipelines to better check and correct legal citations in multilingual cases.
- •For machine translation engineers: Design more reliable translation workflows for legal documents where placement of translation steps has minimal effect on key faithfulness measures.
Tested on one dataset.
Authors
Nguyen Minh Chi, Mo El-Haj, Nguyen Ha Thanh, Dawn Knight, Paul Rayson
Abstract
Cross-lingual legal question answering must retrieve statutes across languages while preventing unsupported legal claims. We introduce a bilingual evaluation suite of 231 Vietnamese--English question--answer pairs from Vietnamese labour law. Of these, 75 are additionally annotated for five challenging legal reasoning phenomena. We evaluate a verifier-guided pipeline that decomposes answers into claims, checks citation reachability and entailment, and corrects citation failures and contradictions. We also introduce six automatic diagnostics for faithfulness to retrieved evidence, covering citations, modality, exceptions, procedures, conclusions, and evidential support. Experiments show that learned-sparse retrieval performs poorly for English-to-Vietnamese retrieval (R@5~=~0.032), whereas dense retrieval reaches 0.358 and slightly outperforms hybrid retrieval. Translation placement has no statistically detectable effect on these automatic diagnostics in our controlled comparison and supporting sensitivity analyses. Verifier-guided correction improves citation preservation by $0.022$--$0.034$ at the system level but produces no reliable gains in the remaining dimensions. Human evaluation further shows that the automatic diagnostics do not fully align with human judgements of answer quality.