Language models controlled more precisely across languages and tasks
Compositional Activation Steering Across Languages and Behaviors with Contextual Gates
Computation and Language
Summary
Getting language AI to do multiple things at once is tricky, especially for different languages. The authors found a way to adjust the AI’s ‘settings’ dynamically depending on what it’s working on, making it better at handling several instructions at the same time. Their method works better than other ways that use fixed settings or simple prompts. It also keeps the AI’s overall knowledge intact and operates more naturally. This helps understand how combining instructions can steer AI behavior effectively.
What this means in practice
- •For multilingual chatbot developers: Control multiple chatbot behaviors simultaneously across languages with adaptive steering for more precise responses.
- •For ai content moderation teams: Adjust models to detect and moderate varied behavioral attributes in multiple languages without retraining the entire model.
Authors
Daniil Gurgurov, Akansh Maurya, Josef van Genabith, Simon Ostermann, Hitomi Yanaka
Abstract
Activation steering has emerged as an effective and precise way to influence the behaviours of language models. However, composing multiple steering directions to control several behaviours simultaneously remains challenging, particularly in multilingual settings and with training-free steering vectors. We address this challenge with contextual steering, jointly training gates that adapt the strength of fixed steering directions to the current hidden state and interactions between interventions. Across a broad range of languages, behavioural attributes, and models, our approach outperforms constant-strength steering and four strong baselines, including prompting. It also better preserves general knowledge and keeps the model's internal computation closer to that of the prompted model. Our analysis further shows how jointly trained gates adapt their contributions during composition, explaining the mechanisms underlying effective compositional steering.