Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation
2026-09-01 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors studied how large language models (LLMs) judge the quality of generated text, like summaries. They found that these models use a two-step process: first, lower layers compare errors locally, and then higher layers combine this info to give a final score. Fine-tuning changes how these steps work but builds on an existing system inside the model, rather than creating a new one from scratch. They tested their ideas using experiments on two specific LLMs and shared their code and data for others to explore.
Natural Language GenerationLLM EvaluatorsFine-tuningAttention MechanismMLP CascadeCausal TracingResidual StreamLogit LensError DetectionModel Interpretability
Authors
Himil Vasava, Ming Jiang
Abstract
LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps, and a four-experiment battery of causal tracing, logit-lens vocabulary projection, and attention-head knockout applied to Themis (Llama-3-8B) and Prometheus (Mistral-7B). Both evaluators implement a structured, coherent evaluation pipeline operating in two stages: below layer 15, attention performs local error comparison and routes the result to the final input position; above it, the MLP cascade integrates the signal and writes the rating, with the decision crystallizing in the residual stream at a sharp late layer (L = 26 on Themis, L = 25 on Prometheus). Furthermore, a base-model control at the same scale (Llama-3-8B) reproduces the routing architecture and crystallization but not the stage separation, isolating the two mechanisms that fine-tuning specifically installs, suppression of below-L15 MLP contribution at the last position and a two-layer advance of the crystallization depth, indicating that fine-tuning sculpts an existing substrate rather than building the pipeline from scratch. We release the source code and data at https://github.com/himil-v/judge-mech