Language models get precise control over following instructions
DIAL: Steerable Language Models with Distilled Instruction-Adherence Levels
Artificial IntelligenceMachine Learning
Summary
Sometimes when you tell a language model to do something, like "keep it short," it’s unclear exactly how short the answer should be. The authors studied how to give models precise, adjustable control over how closely they follow instructions. They created a method called DIAL that teaches models to follow instructions at different levels of detail while keeping answers fluent and natural. This method works efficiently and can generalize to many types of instructions without extra training for each one. They also made a large collection of instructions called SteerIF to help train and test their approach.
What this means in practice
- •For chatbot developers: Adjust chatbot responses to follow user instructions more precisely to get the desired response style or length.
- •For customer support platform teams: Improve automated reply systems by controlling how strictly they follow templates and personalization instructions.
Authors
Thanh Q. Tran, John Russell Himawan, Arun Verma, Kiwan Wong, Wei Xiao, Daniela Rus, Bryan Kian Hsiang Low
Abstract
When a user says "Keep it short", how short should the LLM's response be? More generally, users and applications require different levels of instruction adherence, but prompting does not reliably provide fine-grained control over them. We therefore study "steerable language models" that provide fine-grained numerical control over the degree to which models follow general instructions specified in text. However, existing approaches do not jointly offer fine-grained control, efficiency, and scalability across diverse instructions. To address this gap, we introduce DIAL (Distilled Instruction-Adherence Levels) to achieve all three properties. For fine-grained control, we perform classifier-free guidance (CFG) with a frozen LLM to provide training signals for on-policy distillation of instruction adherence at different levels. For efficiency, we design a lightweight steering adapter that uses a classifier over the LLM's hidden states to compute a closed-form steering edit. The classifier adaptively sets the steering magnitude at each token based on the target adherence level, while accounting for the geometry of the LLM's hidden states to preserve response coherence. For scalability, the classifier is text-conditioned, allowing a shared adapter to support new instructions without per-instruction fitting, and training requires only instruction and request text. Evaluation across multiple LLM families shows that DIAL achieves a competitive balance between concept expression and response fluency, with concept scores increasing monotonically across adherence levels. We introduce SteerIF, an instruction-only dataset of 98, 625 concept, persona, verifiable, and descriptive instructions. When trained on SteerIF, DIAL provides fine-grained control over personas and, when combined with prompting, improves verifiable instruction following.