Debate training methods improve learning with imperfect human judges
Towards a Theory of Learnability via Debate
Machine Learning
Summary
Training AI models by having them debate helps humans decide which answer is better instead of just giving the correct label. The authors show that some learning strategies work better than this debate approach in certain cases. They propose a cooperative version of the debate to understand when learning is possible and also highlight that even a few mistakes by human judges can cause big learning errors. This work is a first step toward understanding how debate-based training actually helps AI learn.
What this means in practice
- •For machine learning engineers: Design training methods that account for human mistakes in judging debates to improve model learning effectiveness.
- •For human-computer interaction teams: Create interfaces that support cooperative debate formats to help humans guide AI learning under noisy judgments.
A theory result. No direct application yet.
Authors
Tosca Lechner, Vinayak Pathak, Sivan Sabato, Ruth Urner
Abstract
Debate has been recently proposed as a training protocol that is hoped to produce more aligned models even as model capabilities scale. The idea is that the human, instead of providing supervision in the form of labels, judges a debate between two models that compete to prove the correctness of opposite labels. It can be shown that functions of a much higher computational complexity can be correctly judged in this way, compared to the complexity of functions that can be directly evaluated using a computationally weak evaluator. This suggests that perhaps debate can be used to teach very complicated functions to models. However, previous work has largely ignored the fact that this is a training protocol, and simply analyzes the game being played between the two debaters for a known judge. We take the first steps towards understanding the learning problem implicit within the protocol, by formulating it as an online learning problem. We show that once we formulate the learning problem precisely, there are cases where the best way to learn given access to a judge is different from running an adversarial debate. This motivates us to define a cooperative version of the protocol, whose learnability we characterize using a combinatorial dimension. We also consider the possibility of the human judge being fallible and occasionally judging the debate incorrectly. We show that a small number of errors in judgement can lead to a large error in learning.