Llms show weaknesses under adversarial cultural value tests in chinese
ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values
Computation and LanguageArtificial Intelligence
Summary
Large Language Models (LLMs) need to understand and respect different cultural values to be safe worldwide. The authors created ABench, a set of tricky questions in Chinese designed to test how well LLMs handle complex cultural values under pressure. They found that most popular models struggle, scoring below 60, revealing weaknesses in their ability to stay aligned with these values. This new benchmark aims to help improve culturally aware LLMs in the future.
What this means in practice
- •For llm developers: Evaluate and improve large language models’ ability to maintain stable cultural value stances under adversarial conditions in Chinese contexts.
- •For multilingual software engineers: Enhance chatbot and AI system safety by using ABench to detect vulnerabilities in handling Chinese cultural values.
Authors
Liping Yang, Yongqi Leng, Jingting Zheng, Yuqi Ren, Shaolin Zhu, Deyi Xiong
Abstract
Ensuring the alignment of Large Language Models (LLMs) with diverse cultural values is critical for their safe global deployment. However, existing alignment benchmarks are predominantly English-centric or confined to simple discriminative tasks. Moreover, they often fail to effectively evaluate model robustness against complex, adversarial inductions within specific cultural contexts. To address these limitations, we propose ABench, a fine-grained, handcrafted benchmark specifically tailored to taxonomized values in Chinese. Grounded in a rigorous hierarchical taxonomy comprising 158 derived value subcategories, ABench consists of 2,895 open-ended questions. These questions employ techniques such as role-playing and logical traps to probe the stability of models' stances in adversarially induced contexts. Furthermore, we propose a weighted automated evaluation framework that utilizes question-specific scoring criteria for nuanced assessment. Extensive experiments on 16 mainstream LLMs reveal that most models exhibit significant vulnerabilities when subjected to sophisticated adversary attacks on values (scoring below 60). This underscores the need for more robust, culturally specific alignment strategies. Our data will be publicly released to promote further research on taxonomized value alignment.