Compact decoder improves when and what AI assistants suggest during chats
When to Assist and What to Suggest: Joint Gating and Memory-Grounded Hint Generation in One Compact Decoder
Computation and LanguageArtificial IntelligenceHuman-Computer Interaction
Summary
Proactive AI assistants try to help by giving hints at the right moments in a conversation. The paper looks at how to improve these assistants by using one smaller model to decide when to help and what hint to give, instead of separate larger models. The authors found that using memory about the conversation is very important for giving good hints. Their single model matches or beats previous setups while being faster. People also rated the single model’s suggestions as helpful and accurate, especially when it uses memory.
What this means in practice
- •For customer support teams: Build chat assistants that decide when and what hints to suggest to users with fewer computing resources, speeding up response times.
- •For virtual assistant developers: Create compact AI models that combine decision-making and hint generation to improve efficiency and maintain quality in interactive dialogue systems.
Authors
A. J. A. Jubayer Talukder, Mohammad Ariful Haque
Abstract
Proactive assistants offer short, memory-grounded hints during a conversation, deciding when to assist and what to suggest. Current systems split this between a gate and a larger generator; we evaluate both together on 28,058 decision points. Stage-wise generation scores are oracle-gate ceilings: behind a real gate, end-to-end exact match is 0.2619 against a ceiling of 0.2971, and end-to-end precision is 0.2387, leaving substantial headroom. A gate pays off through precision and compute, whose saving falls from 4.39x with an 8B generator to 1.38x with a 1B one. The memory profile drives hint quality: removing it costs 24.7 exact-match points (rescaled BERTScore 0.57 to 0.35), while retrieval and a larger generator add nothing on exact match. One 1B decoder can do both, gating from an intermediate layer and generating in the same pass. At its final checkpoint it matches or exceeds the two-model pipelines on automatic end-to-end measures with 1.31-1.76x less wall-clock time than the matched 1B pipeline; across three seeds, its deeper exit ties the separate gate. A human evaluation (1,144 ratings over 32 dialogues) supports the memory effect and rates the single model level with the matched 1B pipeline and below the 8B one. Graded metrics and LlamaPIE's native judge track human ratings more closely than exact match, which we report as a strict lower bound.