AI summaryⓘ
The authors developed three methods to help find the right legal articles in a large collection of Polish laws by attaching special language-model-based tags to each article. They tested these methods against other common approaches using real exam questions for future lawyers, finding that one method (ASCR-H) was best at quickly identifying the most relevant article. However, this advantage mostly works when looking at the very top results and fades when checking more articles, where a faster method (DTF) performs well and costs less. They also found that some common search tweaks did not improve results, and their tagging covers only part of the law collection but always includes the needed articles for their tests.
statutory lawdocument surrogateslanguage-model annotationsrerankingdense retrievallexical retrievalreciprocal rank fusionbenchmark evaluationMcNemar testquery rewriting
Abstract
We present a family of retrieval methods for Polish statutory law built on document surrogates: language-model annotations attached to statutory articles at index time. Three designs occupy different points on the cost-quality frontier. ASCR is a surrogate cascade with reranking; ASCR-H fuses a dense list into that cascade; and DTF replaces both language-model stages with three lexical and dense retrievers, weighted reciprocal rank fusion, and a deterministic re-scoring prior, using no model call before generation. We evaluate all three against fourteen lexical, dense, fused and ablated baselines plus four controls, on 300 questions from the 2024 and 2025 Polish bar and legal counsel entrance examinations (264 with their reference article in the corpus), over 82,508 articles from 1,133 acts. On paired McNemar tests, ASCR-H places the reference provision at rank one significantly more often than every other non-oracle configuration except one of its own ablations (eighteen of twenty comparisons significant in its favour at p < 0.005), reaching 72.3% against 61.7% for BM25 and 52.3% for dense retrieval. The advantage is concentrated at the head and does not survive depth: it is significant at cutoffs of one and five, disappears by ten, and by twenty DTF leads on point estimate (86.0% versus 84.5%) at one ninth the latency and less than half the cost. Ablation attributes 27.6 points of rank-one accuracy to the reranking stage alone. We further report that the ranking advantage does not extend to citation accuracy, where DTF matches the oracle ceiling, and three negative results on lemmatisation, pseudo-relevance feedback and query rewriting. Surrogate annotation covers 27.0% of the corpus but every reference provision in the benchmark, an asymmetry we disclose and discuss. Benchmark, per-question outputs and paired significance tests are publicly available.