Consistency training reduces cue use without teaching when cues are correct
Consistency Training Suppresses Cue Following Without Teaching Cue Selectivity
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Language models sometimes change their answers based on extra hints that don’t actually provide real information. The authors studied methods that teach models to give consistent answers with or without those hints. They found that while these methods do lessen the effect of these hints, they don't help models learn to tell when a hint is actually helpful or not. Instead, the hint’s influence weakens equally whether the hint is right or wrong. This shows that current training methods suppress cue-following but don't improve the model’s understanding of cue reliability.
What this means in practice
- •For nlp engineers: Reduce overreliance on irrelevant hints in language models to improve answer stability when deploying NLP systems.
- •For ai safety teams: Evaluate how mitigating cue-based errors affects model reliability and residual vulnerabilities in deployed AI assistants.
Authors
Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin
Abstract
Language models often change answers because of a cue, a hint such as "I believe the answer is B" that gives no valid evidence. Consistency training (CT) trains a model to answer the same with and without the cue. Four CT methods apply this idea to output tokens (BCT), residual activations (ACT), attention patterns (AttCT) and sampled-answer rates (RMCT). Prior work suggests some learn different solutions. We train all four on Gemma-2-2B-it and Llama-3.1-8B-Instruct with the same data. We test predictions written in advance and inspect the models with a known circuit that carries cues to the answer. All four make the trained cue less effective, and their fixes share three properties. First, correct cues lose about as much influence as wrong ones. Second, the fix is usually large for cues before the answer options, as in training, and much smaller after them. Llama RMCT depends least on position. Third, the circuit heads still carry most remaining cue-induced errors. Making BCT targets depend on the cue's correctness mostly weakens the fix. CT thus weakens the trained cue without teaching models when a cue is right. Evaluations with only wrong cues at the trained position miss the first two properties.