Large language models show varied political leanings based on test setup

Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models

Computation and LanguageComputers and Society

Summary

Large language models are used to provide information, but it's tricky to know their political leanings because test results can be affected by how questions are asked and other factors. The authors created a detailed evaluation method that changes many test settings to see how these affect the models' political scores. They found that most models lean toward the Libertarian-Left, but changes in language, instructions, and answer formats can shift this. Also, smaller models can appear neutral simply because their answers are weak signals instead of truly centrist views.

What this means in practice

  • For nlp engineers: Improve measurement of political bias in multilingual language models by applying robust, multi-dimensional test settings to get clearer political positions.
  • For hate-speech moderation teams: Better tailor hate-speech detection settings by understanding how model size and political role prompts influence classification on sensitive datasets.

Authors

Luka Debevc, Nishan Chatterjee, Antoine Doucet, Senja Pollak, Matej Martinc

Abstract

Large Language Models are increasingly deployed as information intermediaries, yet measuring their political behavior remains fragile because questionnaire results mix model dispositions with measurement artifacts and response-elicitation biases. We introduce a robust Political Compass Test evaluation framework that samples 300 configurations across an eight-dimensional perturbation space varying language, framing, instructions, answer format, option order, and persona wording. We evaluate eight Gemma 3 and Qwen 3 models across 14 languages and three quantization levels, obtaining design-averaged political coordinates with quantified uncertainty. Most models lean Libertarian-Left on average, but instruction phrasing, language, and answer format significantly affect recovered coordinates. Cross-lingual differences primarily reflect coordinate drift rather than distinct cultural reasoning. Reverse-engineering the test also exposes axis-weighting imbalances and the collapse of degenerate responses toward the center, so near-origin estimates for the smallest models can reflect weak signal rather than centrism. Free-text reasoning and chat-then-classify elicitation alter recovered coordinates, and larger models show clearer persona separation, with a specific failure of the Authoritarian-Left persona to move most models in the intended social direction. In downstream tasks, persona effects are modest relative to model size and target group for hate-speech detection, while base and centrist prompts give the highest agreement for topic-level sentiment. Political role prompting therefore has measurable but task- and dataset-specific downstream effects.