User preference steering improves personal assistant responses without retraining
UPSHIFT: User Preference Steering from Human-AI Interaction Feedback Traces
Artificial Intelligence
Summary
When interacting with AI assistants, people have different preferences in how answers are given, but these preferences are often hidden until users ask for changes. The researchers show a way to learn these preferences from how users revise answers, without needing detailed profiles or retraining the AI. Their method, called UPSHIFT, looks at how responses change during these interactions and uses that to guide future answers to fit the user better. Tests with simulated users show that this approach improves how well the AI aligns with individual preferences.
What this means in practice
- •For chatbot developers: Adjust chatbot responses on the fly to better match user preferences without retraining the model or storing full interaction histories.
- •For customer support teams: Improve automated replies by incorporating common user revision patterns across sessions to better satisfy individual users with fewer explicit settings.
Tested on simulated data.
Authors
Jingping Wang, Zhanhao Li, Shao Zhang, Shuyue Hu
Abstract
Personalizing large language model assistants requires adapting their responses to individual users, including the selection of content, reasoning approach, level of detail, and form of presentation. Such preferences are often latent, becoming visible only when users respond to an initial answer and request revisions. We investigate whether these feedback trajectories can be converted into a reusable personalization signal without explicit preference profiles or labels, prompt-time access to interaction histories, or model parameter updates. We introduce UPSHIFT (User Preference Steering from Human-AI Interaction Feedback Traces), a training-free method that compares the hidden representations of initial and feedback-conditioned responses within each session, averages the resulting trajectory displacements across a user's sessions, and injects the aggregated preference direction when answering a new query. The method is motivated by the implicit-preference hypothesis: although an individual conversational transition may reflect topic progression or task-specific correction, changes that recur across sessions retain a component that captures how the user wants responses to be adjusted. We validate the preference-representation capability of UPSHIFT on PrefEval. In evaluations with 40 simulated users, every tested configuration improves preference alignment over the unsteered baseline on held-out topics for both Qwen3-8B and Qwen3-32B; the best configuration increases macro-averaged Elo from 1364.0 to 1605.2 and from 1341.9 to 1591.8, respectively. Matched-user and positive-versus-negative controls further demonstrate that these gains arise from the user-specific preference direction and its learned orientation. These results show that interaction histories can provide compact, reusable supervision for lightweight personalization.