Summary
Evaluating whether language models are fair often looks at one sensitive trait at a time, like gender or race, which can miss problems that happen when traits overlap. The authors created ComFair, a big set of test cases that combine two or more traits and pair them with similar tasks without added traits, to see how this changes model behavior. They tested ten language models and found that some errors appear only when multiple traits are considered, and the severity and type of errors depend on the kind of task and attribute combination. This work helps check if fairness assessments that consider only one trait at a time might be missing important problems.
What this means in practice
- •For llm developers: Test language models for fairness problems involving multiple combined identity traits beyond single-attribute checks.
- •For ai ethics teams: Assess fairness audit tools for their ability to detect subtle, intersectional biases in language model outputs across various tasks.
Authors
Zheng Zhang, Yifan Wu, Yuxiang Zhou, Qi Liu, Qingyang Mao, Ning Li, Weibo Gao, Qiyuan Shi, Zhenya Huang, Baosheng Yu
Abstract
Large language models (LLMs) are increasingly evaluated for fairness, yet most existing benchmarks audit one sensitive attribute at a time. This single-attribute paradigm can miss intersectional harms, where multiple attributes interact non-additively and produce failures that are invisible under marginal evaluation. We introduce ComFair, a configurable benchmark that converts existing fairness data into 479,066 controlled Dual- and Multi-attribute instances and pairs every augmented instance with its task-equivalent source. Across Recognition, Selection, Continuation, and Conversation, this pairing supports a unified audit of performance or bias-score shifts, how often added attributes introduce new errors or mask existing ones, and model-ranking stability. Experiments on ten LLMs show task- and subgroup-dependent sensitivity: discriminative tasks expose both transition directions even when aggregate changes are modest, generative tasks exhibit larger and directionally asymmetric shifts, and rankings vary with attribute composition. ComFair therefore provides a scalable controlled sensitivity audit---rather than a model of the natural joint distribution of identities---for testing the robustness of fairness-benchmark conclusions beyond single-attribute settings.