ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models
2026-08-20 • Computation and Language
Computation and Language
AI summaryⓘ
The authors explain that current methods for making language models forget harmful information are not good enough because they treat facts separately and don't consider how the same idea can be used both safely and dangerously. They introduce a new concept called 'dual-use concepts' which are ideas that can be harmful or helpful depending on context. To test unlearning more realistically, they create a benchmark named ConceptGuard that checks if models can forget harmful uses of a concept while keeping the helpful ones. Their tests show existing methods struggle with this and highlight challenges in balancing forgetting bad behaviors while keeping useful knowledge.
Large Language ModelsUnlearningDual-use conceptsBenchmarkFactual recallContextual separationConcept-level controlForgetting-utility trade-off
Authors
Sahil Kale, Ian Harris
Abstract
Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called unlearning, yet existing methods and benchmarks fail to evaluate this capability completely. Current approaches rely on disjoint forget and retain sets composed of independent facts, and measure success using simple and direct factual recall. This framing fails to capture a key requirement of unlearning, namely the ability to eliminate harmful behaviors while preserving benign and beneficial knowledge. We argue that effective unlearning must operate at the level of concepts, ensuring complete removal of unsafe applications while maintaining their correct and useful usage, thereby achieving conceptually meaningful and complete unlearning. To better evaluate unlearning techniques from such a practical viewpoint, we introduce the notion of dual-use concepts: concepts that can be used in both harmful and benign contexts. Building on these concepts, we construct a benchmark called ConceptGuard where forget and retain sets are explicitly complementary in concept usage. Our benchmark uniquely enables unlearning to be explored and gauged at the level of concepts, instead of sparse facts, and evaluation is intent-sensitive with the goal of maximizing contextual separation to promote safer behavior. We demonstrate that current unlearning techniques perform poorly under this setting, showing weak contextual separation alongside poor performance in ROUGE and concept-level metrics. Our results reveal strong forgetting-utility trade-offs, limited gains in contextual sensitivity, and poor consistency in concept-level control across methods, and provide ideas for unlearning approaches that better align with real-world safety requirements. Our dataset is publicly available.