Correct harmful training data instead of deleting to reduce errors
Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Fine-tuning language models on bad data, like harmful medical advice, can cause them to give wrong answers even on unrelated questions. The authors found that instead of deleting the bad data, correcting it with better answers helps reduce these mistakes more effectively. They tested this on multiple models and showed that replacing bad advice with correct advice works better than just removing it. Also, training on corrected data helps realign a misbehaving model more than training on generic data.
What this means in practice
- •For machine learning engineers: Improve model alignment by fine-tuning on corrected harmful training examples rather than deleting them.
- •For healthcare ai developers: Reduce risks of harmful advice from medical AI systems by replacing faulty training instances with corrected ones.
Authors
Jacob Epifano
Abstract
Fine-tuning a language model on a narrow set of harmful demonstrations, such as bad medical advice, can make it broadly misaligned on unrelated questions, a phenomenon known as emergent misalignment (EM). The usual defense is to find the offending rows and delete them, but a row locator failed our held-out test and deleting rows helps less than expected. We ask a different question: given a fixed set of poisoned rows, is it better to correct them than to remove them? We fine-tune Qwen2.5-14B-Instruct on a mixture of bad medical advice and benign chat data, select a quarter of the poison rows in advance, and either delete them or replace each with a corrected answer to the same prompt, keeping everything else the same. Replacing the rows cuts the EM rate by about a third and improves answers on held-out medical questions, while deleting the same rows has little measurable effect. The advantage is larger when half the poison rows are corrected, and it holds on a second base model and a second misaligned model organism. The content of the replacement appears to matter: paraphrasing the rows while keeping their bad advice shows no clear benefit, and the correct answers distributed with the dataset appear to do about as well as our rewriter's. Realigning an already-poisoned model with further fine-tuning is known to work, but which data does the work has not been compared directly. We find that a short round of training on corrections beats the same amount of training on generic chat data, that corrections on other medical prompts do roughly as well as corrections of the poisoned prompts themselves, and that instructing the correction writer to model a careful, harm-avoiding assistant adds no measurable benefit over plain corrections. In the settings we tested, correcting harmful training data reduces EM more than deleting it.