Llms learn when to stop gathering evidence before closing investigations
Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Investigations often need to decide when they have enough evidence to conclude, which is different from just answering questions. The authors show that large language models (LLMs) usually say they have enough evidence even when they don't, especially without training. They created a dataset of real-world investigation cases and taught a model to better recognize when to end an investigation based on evidence. After training, the model became much better at knowing when to stop and avoiding overconfident conclusions.
What this means in practice
- •For incident investigation teams: Support automated review systems that decide when enough evidence is gathered to close safety or defect investigations accurately.
- •For it operations teams: Use trained language models to help determine when to conclude production outage investigations based on the evidence available.
- •For customer service automation teams: Enhance chatbot systems that investigate user issues to avoid premature case closures and request missing details when needed.$Commercial implications: This paper enables building more reliable automated customer support systems that confidently decide when they have sufficient information to resolve issues.
Authors
Tingzhu Bi, Ping Wang, Meng Ma
Abstract
Accident, defect and outage investigations end with a decision that ordinary question answering never faces: whether the evidence gathered so far is enough to close the case. We study this decision for LLM investigators, which request evidence from a case file, revise their hypotheses, and either close the case with a conclusion grounded in what they read or leave it open and name what is missing. This judgment does not come with capability: an untrained 9B model overstates its evidence in 97% of its answers, and a frontier model that identifies the right cause in 84% of cases still overstates in 91% and closes 17 of the 41 cases whose official finding is "cause undetermined". Measuring it is also non-trivial: the source of a case largely predicts its label, and a rule that reads only the source reaches 83.0 balanced accuracy on our test cases. We therefore evaluate closure with three tests: closure accuracy, reported against this rule and within each source; evidence dependence, which removes the grounds of a conclusion and checks whether the model stops closing; and conclusion and gap quality, a judged checklist of what the model asserts and what it says is missing. We build Nautil, 731 audited cases from aviation, rail, maritime, chemical-safety and vehicle-defect reports and production server incidents, with teacher trajectories, an out-of-distribution test set and counterfactual evidence versions. Fine-tuning a 9B model on these trajectories makes its closures follow the evidence: removing the grounds lowers its closure rate by 26 points relative to a matched control, overstatement falls from 97% to 35%, and correct, non-overstated conclusions rise from 3% to 43%. Reinforcement learning that rewards only the closure decision then raises balanced accuracy from 69.2 to 83.3, on par with the teacher, and within-source accuracy from 60.4 to 74.1, at some cost in evidence dependence.