Delayed Countdown method improves validation of AI document review
The Delayed Countdown Method for Validating Review-Oriented AI Workflows in eDiscovery
Information Retrieval
Summary
Checking how well AI helps review legal documents can be tricky and debated. The authors studied why this is confusing and suggested a new way called the Delayed Countdown method. This method fits easily into current workflows and software, works with any stopping rules, and gives reliable statistics on how effective the review is. They also show examples, handle tricky cases where some results are impossible, and hint at ways to reduce review costs.
What this means in practice
- •For legal technology teams: Integrate valid statistical validation of AI-assisted document review tools to improve confidence in review outcomes and compliance.
- •For data quality assurance teams: Use the delayed countdown method to reliably measure review effectiveness when validating document classification processes in other regulated industries.
Authors
David D. Lewis, Tara Emory, Lenora Gray, Mark Noel, Jeremy Pickens, Eugene Yang
Abstract
Review-oriented workflows have become the dominant approach to AI-assisted document review in electronic discovery (eDiscovery). How to validate the effectiveness of review-oriented workflows has, however, remained a confusing and controversial topic in technology-assisted review (TAR) in the law. We examine the sources of this controversy, and propose a simple and practical evaluation approach for review-oriented workflows. Our Delayed Countdown method is easily incorporated into existing review-oriented workflows and software, is compatible with arbitrary stopping rules, and produces valid frequentist confidence intervals on effectiveness. We explain the conceptual foundations of the method, provide example computations, discuss the handling of logically impossible recall values, and suggest future work on decreasing review costs.