Chatbots give stable judgments on presidents actions and reforms
Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms
Computation and LanguageComputers and Society
Summary
Chatbots are often asked about political decisions and policies, but their answers need to be consistent to be trustworthy. The authors tested five popular chatbot setups over a month, asking them about past presidential actions and possible reforms, and found they mostly agreed on whether these were good or bad and how they ranked overall. They also discovered that answers were more consistent when given on the same day than across days, except when the chatbot service was updated. The authors are sharing a large set of these chatbot judgments for others to use.
What this means in practice
- •For public policy analysts: Use chatbot judgments as a reliable input to evaluate long-term effects of government actions and reforms.
- •For social media monitoring teams: Monitor shifts in chatbot responses that may signal service changes, helping detect subtle model updates affecting output consistency.
Authors
Oleg Vasilyev
Abstract
Chatbots are increasingly consulted about public affairs, and their judgments of policies are useful only if they are robust. We measure this robustness in the setting of an ordinary user, through consumer chat interfaces. For 30 days, five chatbot configurations (Mistral, Gemini Flash, Claude Sonnet, and Claude Opus at two effort settings) were asked twice a day to score the long-term effect on the United States of 40 actions of the last four presidential administrations and of 15 hypothetical reforms, each with a confidence and a short comment. The judgments are robust: all configurations agree on the sign and the ranking of the administrations' mean scores, and two different configurations asked on the same day give opposite signs to the same action in about 4% of cases. Responses from different days differ more than same-day responses, but the difference grows slowly over the month, except where the service itself changed. We identify three such changes, which shifted the judgments while leaving little or no trace in the interface. Treating the comments as a retrieval task, we find them specific to each item and consistent across runs and chatbots, with lexical and embedding similarities agreeing. We release the dataset of 16,500 judgments.