Language models face challenges judging their own skill with noisy feedback
LADDER: Diagnosing Domain-Level Self-Assessment Under Noisy Feedback
Computation and Language
Summary
Language models need to know when they are right to work well with people. The authors studied how these models judge their own performance when the feedback they get is sometimes wrong or confusing. They found that even if models report feedback accurately, this doesn’t mean they truly understand how good they are or how well they will do later. This shows it’s important to check a model’s self-judgments against actual results to trust them when working with humans.
What this means in practice
- •For ai system engineers: Improve human-AI collaboration by evaluating language model self-assessments against real task performance to avoid reliance on misleading feedback.
- •For customer support teams: Better identify when AI chatbots are uncertain or unreliable by comparing their self-reported confidence to actual conversation outcomes.
Authors
Zhenshuai Yin, Jianshu Liang, Xuyue Xiong, Yongxin Guo, Mingda Zhang, Zhongze Luo, Haoyue Liu, Lianmin Zhang, Xiaoying Tang
Abstract
Knowing when they are reliable is essential for language models to support human-AI collaboration. Yet interaction feedback is not always complete or correct, and may mislead models' understanding of their own capabilities. We introduce LADDER to study how language models assess themselves under unreliable feedback. Our findings show that models can accurately report supplied performance information while still struggling to form trustworthy self-assessments from interaction experience. Apparently accurate assessments can obscure capability differences and do not necessarily predict future task performance. These findings suggest that trustworthy self-assessment should be evaluated against observed task performance and its predictive value. Our study provides empirical evidence of the limits of model self-knowledge and a basis for evaluating capability awareness in reliable human-AI collaboration.