Conservative bandits ensure safe learning from user preferences
Conservative Dueling Bandits: Preference Learning under Baseline Constraints
Machine LearningArtificial Intelligence
Summary
Some systems learn from comparing pairs of options but can sometimes show users two bad choices while exploring. The authors study a careful way to learn that always keeps user experience close to a trusted baseline, so users rarely see worse options. They created new methods that guarantee safety throughout learning and still improve over time, even when the best option is unknown. Their experiments show these new methods keep safety while learning moderately well.
What this means in practice
- •For online recommendation teams: Maintain user satisfaction by safely learning preferences without exposing users to much worse options during exploration.
- •For interactive language model developers: Fine-tune language models from human preference data while ensuring the displayed choices remain competitively good throughout training.
Authors
Rohan Deb, Mohammad Ghavamzadeh, Arindam Banerjee
Abstract
Contextual dueling bandits learn from pairwise comparisons, the same feedback used to fine-tune language models from human preferences. Standard algorithms explore aggressively and can show users two poor options. We study conservative contextual dueling bandits, which require that, at every round, the cumulative score of the better displayed item stay above a $(1-α)$ fraction of a trusted baseline's. An item's score is its probability of beating the best available item, so the constraint depends only on utility differences and needs no absolute rewards. This also changes how safety can be certified: we show that transferring the safety test of the conservative bandit algorithm CLUCB unchanged can incur linear regret, even when the baseline's utilities are known exactly. For linear utilities, we develop MaxInP-BC, which certifies scores through utility gaps and, when it must fall back, pairs the baseline with an informative partner so that fallback rounds still drive learning. It is safe with high probability at all times, with regret $\tilde{O}(d\sqrt{T}/κ)$ plus an additive cost of conservatism that does not grow with the horizon. For general utility classes, we develop BTIGW-BC, which turns the guarantee of an online logistic-regression oracle into cumulative safety certificates. It is safe at all times, makes finitely many fallbacks, and has sublinear regret whenever the oracle's logistic regret is sublinear. In experiments, both satisfy the constraint, which their unconstrained counterparts often violate, at a moderate cost in regret.