Evolutionary method finds harder cases for model evaluation preserving answers
HARDEN: Constrained Evolutionary Search for Harder, Answer-Preserving Evaluation Cases
Artificial Intelligence
Summary
Many language models are tested using examples that might be too simple compared to real-world challenges. The paper introduces HARDEN, a method that changes existing test questions to make them more difficult while keeping the expected answers the same. This method uses an evolutionary search that carefully keeps the tasks realistic and meaningful. When tested on several datasets and large models, HARDEN found harder examples that lowered model accuracy significantly, showing it can create better evaluation cases.
What this means in practice
- •For model evaluators: Generate more challenging test cases that still keep correct answers to better assess model capabilities in complex real-world scenarios.
- •For enterprise ai teams: Improve robustness assessment of language AI used in business by testing models on evolved, harder input cases that reflect realistic difficulties.
Authors
Aditya Kumaran, Rahul Singhal, Karime Maamari, Amine Mhedhbi, Pradyumna Tambwekar
Abstract
Language models are often evaluated on curated benchmarks that underrepresent the complexity of enterprise deployments. We introduce HARDEN, a constrained evolutionary search method to adapt the input of existing evaluation cases into more challenging variants while keeping their expected outputs fixed. HARDEN searches along generated domain-specific complexity axes while enforcing feasibility constraints such as preserving task semantics, realism, and execution validity. Across FinQA, PubMedQA, and ContractNLI and three Qwen3.5 model scales (35B-A3B, 122B-A10B, and 397B-A17B), HARDEN reduces task-model accuracy by 22.7% on average and by up to 49.9% relative to single-pass baselines using the same feasibility checks. These results show that evolutionary search can produce substantially harder valid evaluation cases.