Large language models struggle to stick to user perspectives accurately
Beyond Factual Accuracy: A Benchmark for Evaluating Perspective Alignment in LLMs
Computation and Language
Summary
Sometimes, smart AI text generators give answers that are factually correct but mix in viewpoints the user did not ask for, making it hard to get exactly what they want. The authors created a new test called PerspectiveBench to measure how well these models follow specific perspectives without adding extra views. They found that current models include requested information, but they also add other unwanted perspectives. Even when given source documents, the models still have trouble staying focused on one viewpoint.
What this means in practice
- •For chatbot developers: Measure and improve how chatbots stick to user-specified viewpoints when generating responses.
- •For customer service teams: Ensure AI agents maintain consistent perspectives tailored to customer inquiries without mixing conflicting information.
Authors
Sunghoon Kim, Sangyeop Kim
Abstract
Large Language Models often generate factually correct responses that fail to maintain user-requested perspectives, blending multiple perspectives and forcing users to filter through unwanted information---a perspective control failure that factual accuracy metrics cannot detect. We introduce PerspectiveBench, a benchmark quantifying perspective alignment by constructing scenarios where multiple valid reasoning paths coexist, and propose the Perspective Alignment Score (PAS) to measure adherence to specified perspectives. Experiments reveal that state-of-the-art models successfully include target content but fail to exclude non-requested perspectives. Providing documents through RAG improves factual grounding but does not resolve perspective drift, as models prioritize comprehensive coverage over strict user constraints. Code and data are available at https://github.com/sunghoon014/PerspectiveBench.