Managing language model errors improves autonomous system decisions

Maximizing LLM Utility While Managing LLM Fallibility

Robotics

Summary

Large language models (LLMs) can help computers make plans and predictions, but sometimes they make mistakes. The authors explain how systems can watch how well these models are doing and change how much they trust their advice over time. By doing this, computers can use the models’ help while reducing the chances that their errors cause problems. The authors also discuss ways to stop the worst mistakes from causing damage, either by independently checking outputs or by keeping risks within limits.

What this means in practice

  • For autonomous system developers: Adjust autonomous system trust in language model outputs based on ongoing reliability estimates to reduce error impact during decision making.
  • For software reliability engineers: Incorporate classical reliability estimation methods with large language model components to better manage software faults in complex systems.

A survey. It maps existing work.

Authors

Ekta U. Samani, Christopher G. Atkeson

Abstract

Large language models (LLMs) can supply autonomous systems with plans, predictions, and judgments, but these contributions are sometimes wrong. This review asks how a system can maximize an LLM's utility while managing its fallibility. Our answer is to adapt the system's reliance on the LLM as evidence about the LLM's performance accumulates. Systems built from imperfect components, such as noisy sensors and faulty software, have long worked in this way, and the classical methods they use can be applied to LLM-based systems as well. A system can change its reliance on an LLM in several ways, depending on the LLM's role and on the other components in the system. Each change needs an estimate of the LLM's reliability, and the LLM's role and the kind of contribution it makes determine what to estimate. Classical methods from decision theory, estimation, and online learning then turn these estimates into reliance decisions. These decisions can reduce how often LLM errors affect the system. However, some errors matter even when they are rare. An independently verified mechanism can prevent such errors from being executed. Without one, adapting reliance using reliability estimates can still keep their risk within a bound. Studies on adapting reliance on LLMs are emerging but limited, and the reviewed comparisons suggest that adapting reliance can improve system performance.