Ethical framework guides ai care through user vulnerability situations
Situating Vulnerability: An Ethical Governance Framework for Anthropomorphic AI Interaction
Computers and Society
Summary
Some AI chatbots act like caring friends, remembering personal details and using emotional language. This can help some users but might harm others. The authors say that rules blocking or limiting these caring responses don’t work well because they don’t consider the ongoing relationship or the different ways people can be vulnerable. They suggest focusing on the situation causing vulnerability, not on labeling groups of users, and offer a framework to guide AI responses more ethically.
What this means in practice
- •For ai product managers: Design chatbot policies that adapt to user vulnerability situations instead of applying uniform or group-based restrictions to improve ethical AI behavior.
- •For customer service platform teams: Create interaction guidelines for AI agents that calibrate empathic responses based on identified vulnerability situations to reduce risks of harm during user conversations.
A position paper. It proposes an approach and reports no results.
Authors
Hanhui Xu, Jiacheng Ji, Yuhan Wang, Haoan Jin, Jaiming Luo, Mengyue Wu
Abstract
Large language models increasingly address users in the register of care: they remember personal details, adopt a first-person emotional voice, and sustain relationships across many sessions. This anthropomorphic mode of interaction generates risks that output-level metrics capture poorly, because the same empathic utterance may console one user, entrap a second, and consolidate a delusion in a third. Governance has responded in two ways. Uniform restriction refuses sensitive requests or suppresses empathic phrasing across all outputs; group-targeted protection profiles vulnerable populations and applies stricter or prohibitive rules to them. This article argues that both fail. Uniform restriction mislocates harm in the lexical form of responses rather than in the trajectory of a relationship; group-based protection depends on a list that can never be complete, treats a fluid property as a static attribute of persons, and converts protective attention into stigmatizing labels. Drawing on philosophical work that treats vulnerability as universal, relational, and differentiated, we propose that the unit of ethical governance should be the vulnerability situation rather than the vulnerable group, develop a typology of five situations - physical, emotional, cognitive, autonomy, and relational - and specify seven principles into requirements that guide response calibration without diagnosing or persistently classifying users. The framework relocates AI safety from output control to relational safety.