Evaluation of large language models on China AI content rules
Benchmarking LLM Compliance with China AI Generated Content Regulations
Computation and LanguageMultiagent Systems
Summary
Large language models (LLMs) generate content but can sometimes produce material that breaks rules. The authors tested 20 big LLMs using thousands of questions related to Chinese content laws to see how well they follow these rules. They found that many international models also comply well with Chinese regulations, especially on politically sensitive topics. The study provides a benchmark to check if LLMs meet China’s legal requirements in a consistent way.
What this means in practice
- •For ai compliance teams: Use the benchmark framework to test if language models meet China’s AI content regulations across multiple compliance dimensions.
- •For multinational ai developers: Assess international models’ compliance with China’s legal content standards to adapt their products for the Chinese market.
Authors
Chenrui Cui, Hongye Fang, Lisha Song, Weichao Chen, Yue Zhu, Gang Xu
Abstract
The widespread adoption of LLMs has led to escalating content compliance risks. Prior works have contributed to addressing these risks in the English context, downplaying the complexity of Chinese language content. This paper follows China's current AI-Generated content compliance requirements and provides evaluation results on 20 notable LLMs, offering insight into China's regulatory landscape. We design a novel framework to assess the compliance and refusal rates with 2303 questions spanning six distinct dimensions, including 203 self-constructed constitutional questions. The framework employs several judges to generate verdicts independently based on their hierarchical alignment memory. Our findings show that international models also exhibit high levels of compliance despite the use of standard Chinese questions, and the main differences may stem from dimensions closely related to ideological alignment. We establish a regulatory benchmark that enables the global AI community to evaluate both Chinese and non-Chinese LLMs under a unified set of legally grounded compliance requirements.