Personalization improves helpfulness of cybersecurity assistant answers
Evaluating the Impact of Personalization in Conversational Cybersecurity Assistants
Cryptography and Security
Summary
People often ask computer programs for cybersecurity advice, but many don’t follow the recommendations. The authors studied different ways to personalize answers to make them easier to understand and more motivating. They found that tailoring answers based on ongoing conversations works best at helping users trust and act on the advice. Their tests showed that automated evaluations with large language models matched human opinions, making it easier to test improvements before real user studies. This suggests that using conversation history can make cybersecurity assistants more effective.
What this means in practice
- •For cybersecurity tool developers: Enhance chatbot advice by personalizing cybersecurity answers based on user conversation history to increase helpfulness and compliance.
- •For customer support teams: Use behavior-driven personalization in automated assistants to provide clearer and more motivating security guidance to users.
Authors
Lea Duesterwald, Anika Jain, Shreya Kochar, Norman Sadeh
Abstract
Users increasingly turn to Large Language Models to answer a variety of questions, including cybersecurity questions. We study how personalization strategies can help improve the effectiveness of answers to questions asked to an LLM-based cybersecurity assistant. Beyond accuracy, we focus on the understandability, actionability and, most importantly, motivating power of answers, given how often users fail to follow cybersecurity recommendations. Specifically, we investigate four personalization strategies, ranging from static user profiles to interaction-history-based personalization, using a corpus of 1,045 real-world cybersecurity questions and a 7-day deployment involving 57 participants and 1,066 user questions. Across both a large-scale automated LLM-based evaluation and human evaluation, conversation-based personalization is consistently favored in comparative ratings of perceived helpfulness and likelihood of following security advice. Importantly, the relative trends observed in the LLM-based evaluation align with those obtained from human evaluation, suggesting that LLM-based evaluation can provide a scalable mechanism for comparing personalization strategies before costly user studies. These results indicate that behavior-driven personalization is a promising direction for LLM-powered cybersecurity assistants and highlight the value of combining LLM-based and human evaluation when studying personalized language-model systems.