Deterministic Fuzzy Triage for Legal Compliance Classification and Evidence Retrieval
2026-03-08 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explored a clear and repeatable way to help legal teams sort contract clauses using a machine learning model called a deterministic dual encoder. They trained this model on legal datasets to detect compliance with rules like HIPAA and achieved strong performance in identifying important clauses and compliance status. Their approach divides results into three groups: clearly compliant, clearly non-compliant, and those needing human review, aiming to limit automatic errors to under 2%. This method offers a simpler and more explainable alternative to complex AI models, making legal evidence review easier to understand and audit.
deterministic dual encoderRoBERTacosine similarityACORD benchmarkCUAD datasetNDCGAUCF1 scorefuzzy triage bandslegal compliance
Authors
Rian Atri
Abstract
Legal teams increasingly use machine learning to triage large volumes of contractual evidence, but many models are opaque, non-deterministic, and difficult to align with frameworks such as HIPAA or NERC-CIP. We study a simple, reproducible alternative based on deterministic dual encoders and transparent fuzzy triage bands. We train a RoBERTa-base dual encoder with a 512-dimensional projection and cosine similarity on the ACORD benchmark for graded clause retrieval, then fine-tune it on a CUAD-derived binary compliance dataset. Across five random seeds (40-44) on a single NVIDIA A100 GPU, the model achieves ACORD-style retrieval performance of NDCG@5 0.38-0.42, NDCG@10 0.45-0.50, and 4-star Precision@5 about 0.37 on the test split. On CUAD-derived binary labels, it achieves AUC 0.98-0.99 and F1 0.22-0.30 depending on positive-class weighting, outperforming majority and random baselines in a highly imbalanced setting with a positive rate of about 0.6%. We then map scalar compliance scores into three regions: auto-noncompliant, auto-compliant, and human-review. Thresholds are tuned on validation data to maximize automatic decision coverage subject to an empirical error-rate constraint of at most 2% over auto-decided examples. The result is a seed-stable system summarized by a small number of scalar parameters. We argue that deterministic encoders, calibrated fuzzy bands, and explicit error constraints provide a practical middle ground between hand-crafted rules and opaque large language models, supporting explainable evidence triage, reproducible audit trails, and concrete mappings to legal review concepts.