Dynamic evaluation reveals flaws in language model safety tests
SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation
Cryptography and SecurityMachine LearningSoftware Engineering
Summary
Measuring how safe and private large language models are usually relies on fixed test sets, but these can miss important problems because they don’t change or adapt. The authors created SSP-Bench, a system that generates new test questions on demand while keeping things accurate and relevant. This helps reveal weaknesses that static tests miss, showing that models can fail when questions are rephrased differently. Their work suggests that ongoing, flexible testing is needed to better understand model safety and privacy.
What this means in practice
- •For ai safety teams: Generate fresh, domain-consistent tests dynamically to better evaluate model safety and catch failures missed by static tests.
- •For chatbot developers: Continuously assess and improve chatbot responses for security and privacy by using dynamically generated, validated evaluation instances.
Authors
Fatih Deniz, Yazan Boshmaf, Issa Khalil
Abstract
Evaluation of large language models (LLMs) for safety, security, and privacy (SSP) relies heavily on static benchmarks, which suffer from score saturation, data contamination, and aggregation artifacts, and fail to capture sensitivity to linguistic variation. As a result, models that perform well on fixed test sets often fail under semantically equivalent rephrasings. We introduce SSP-Bench, a dynamic benchmarking framework that generates evaluation instances on demand while preserving domain consistency. The framework ensures label validity through externally grounded sources, enforces scope via service-specific validation, and calibrates difficulty using a multi-model steering panel. Benchmark construction is formulated as a multi-objective optimization problem over difficulty, separability, novelty, and diversity. Across 24 models and four SSP services, SSP-Bench reveals systematic failures of static evaluation, including near-zero correlation in safety rankings due to construct mixing, strong safety--over-refusal coupling, and hidden within-family regressions. These results show that static benchmarks can misrepresent model behavior, motivating dynamic, deployment-relevant evaluation.