Evaluating RAG for French immigration law: a benchmark and baseline study
2026-07-27 • Information Retrieval
Information RetrievalArtificial Intelligence
AI summaryⓘ
The authors studied how to improve AI systems that help with international recruitment laws in France, which are complex and not covered by current legal AI tests. They created a new public benchmark to check how well AI can recommend permits, find required documents, and identify legal references. By testing two versions of a language model with and without retrieval tools, they found that adding retrieval helps the AI give better and more accurate advice, especially about permit types. Their work shows that combining memory with document search improves legal administrative help and suggests more research on mixing these approaches.
international recruitmentlegal AI benchmarkpermit recommendationdocument retrievallegal citationparametric LLMretrieval augmentationQwen modelsadministrative guidance
Authors
Annia Abtout, Julien Delaunay, Monika Ewa Rakoczy
Abstract
International recruitment in France requires navigating a layered legal framework absent from existing legal AI benchmarks. We present a publicly available benchmark and first comparative evaluation for this domain, covering permit-type recommendation, required-document retrieval, and legal citation coverage. Comparing a parametric LLM baseline against dense retrieval augmentation at two model scales (Qwen3.5-9B and -27B) on 52 annotated synthetic profiles, we find that retrieval improves administrative guidance at both scales, most notably permit-type accuracy. Our results confirm that retrieval grounding is important for more reliable administrative guidance in this domain, and motivate further investigation of hybrid retrieval strategies.