Instruction hierarchy improves by learning from goal conflicts
Improving Instruction Hierarchy by Learning from Goal-Goal Conflicts
Artificial Intelligence
Summary
Large language models (LLMs) need to follow instructions that have different importance, for example obeying system policies more than user requests. The authors found an overlooked type of conflict between two important goals and showed that training LLMs to handle these conflicts makes them follow instructions better. They improved two different LLMs by teaching them with examples of these goal conflicts, helping the models handle real-world tasks better without hurting performance on simple tasks. This work suggests adding these kinds of conflicts to future training datasets for more reliable behavior.
What this means in practice
- •For software engineers: Improve chatbot reliability by fine-tuning LLMs on examples of conflicting user and system instructions to better resolve conflicts.
- •For ai operations teams: Enhance safety of deployed language models by incorporating goal-goal conflict training data to prevent unintended policy breaches.
Authors
Sizhe Chen, Sanjay Kariyappa, Liwei Jiang, Chong Xiang, Jihye Choi, Chaowei Xiao, G. Edward Suh
Abstract
The recent incident in which OpenAI agents autonomously compromised Hugging Face's infrastructure illustrates the importance of instruction hierarchy (IH): an LLM should follow lower-priority instructions, such as user requests, only when they do not conflict with higher-priority instructions, such as system policies. IH can be effectively improved by fine-tuning LLMs on simulated examples of instruction conflicts, whose diversity is key to teaching LLMs to properly handle conflicts at test time. However, our decomposition of a task into goals and constraints reveals a conflict type -- goal-goal conflicts -- that is absent from existing IH training sets from OpenAI and NVIDIA. We show the importance of this conflict type: fine-tuning Qwen3.5-9B and Gemma4-12B-it on examples of this conflict type significantly improves IH on a wide range of real-world tasks unseen in training; adding these examples to existing IH training sets for fine-tuning also gives better IH adherence. The IH gains come with trivial mean utility changes on no-conflict tasks across eight benchmarks. Our work advocates including this new conflict type in IH training, even if these examples do not reflect typical LLM use.