Language models improve judgment alignment with human preferences
PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators
Computation and LanguageArtificial Intelligence
Summary
It is hard to check if language models that judge other language models agree with humans. Instead of trying to match exact scores, the researchers look at whether the model and human preferences agree about which behavior is better. They made a method called PADMÉ that uses smaller models to create data for testing these judgments without needing humans. Their tests show PADMÉ increases alignment with human opinions from 73% to 85%. This helps better understand and improve how language models evaluate agent behavior.
What this means in practice
- •For language model developers: Use PADMÉ to create reliable preference-based evaluation data without costly human labeling for agentic model behaviors.
- •For chatbot product teams: Improve chatbot evaluation by comparing preference alignment of model judgments to human opinions using PADMÉ-generated data.$Commercial implications: Enables development of better evaluation tools to improve chatbot trustworthiness and quality for commercial deployments.
Authors
Cheng Chang, Yining Mao, Peng Qi
Abstract
Language models are frequently employed to evaluate other language models. An LM evaluator scoring agentic behaviors across multiple criteria is valuable, provided that its decisions align with human judgment. We call the problem of evaluating this alignment Meta-Evaluation. Tackling it directly is difficult: collecting human data is expensive, absolute scoring is hard to align, and using an LM meta-evaluator recurses the question of trustworthiness. We adopt a reformulation of meta-evaluation as a preference judgment problem: rather than comparing human and LM evaluator scores of a trajectory, we ask whether their implied preferences align. Building on this, we introduce PADMÉ, a data synthesis method that generates reliable criterion-based meta-evaluation data for agentic settings. PADMÉ uses only small language models, requires no human involvement during evaluations, and operates under a low computational budget. We build a prototype of PADMÉ and synthesize a dataset of 1,000 samples across four agentic domains and three evaluation criteria. Human validation on a 150-sample subset demonstrates that PADMÉ improves agreement with human judgment from 73% to 85% over a naive baseline. Meta-evaluating 25 common models with our dataset demonstrates the correlations between evaluation performance and scoring granularity, leniency, and model size, among other factors.