Mapping and controlling language model preferences by adjusting hidden states
Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs
Machine Learning
Summary
Sometimes language models already know how to give certain answers but don't do it consistently. The authors looked inside the model’s thinking process to find what helps or blocks the preferred responses. They created a way to pinpoint and adjust these internal parts without changing the main model itself. This method achieves similar results to existing training approaches but works by tweaking the model’s internal states during use, making it easier to control.
What this means in practice
- •For machine learning practitioners: Control model outputs by adjusting hidden states during inference without retraining or modifying base weights, enabling flexible preference expression.
- •For software engineers deploying ai: Enable or disable behavior modifications on demand in language models by switching small controllers that adjust hidden states, improving runtime control.
Authors
Fansheng Zhang, Shengran Guo, Zexiao Wang, Liang Yuan, Jiyuan Chen, Ruikun Luo
Abstract
In many settings, post-training need not create the target behavior from scratch: the base model can already produce it, but not reliably. This shifts part of preference alignment from capability acquisition to behavioral expression. We ask how a specified preference is represented in native model computation, what prevents target-supporting computation from reliably dominating generation, and whether this structure can directly guide control. We introduce Residual Competition Maps (RCMs), which map a behavioral preference onto signed causal effects of native residual computation. Across preference domains, RCMs reveal coexisting target-supporting and target-competing effects, input-dependent component roles, and cases where a single native-component intervention reverses the preference outcome. DPO substantially reorganizes these effects and can weaken opposition without guaranteeing its removal. We then propose Direct Hidden-State Alignment (DHSA), which treats inference-time hidden states rather than base-model weights as the direct adaptation space. RCM-guided Causal Activation State Transition (CAST) implements DHSA through local state interventions at a small number of preference-relevant interfaces while freezing the base model. With only 256-16,384 controller parameters, CAST reaches DPO-competitive operating points across three preference domains, can complement DPO-trained models, and can be enabled or removed at inference time.