Automated essay scoring with large language models lacks rubric control
Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base Models
Computation and Language
Summary
Automated systems using large language models (LLMs) can score essays, but these systems struggle to independently evaluate different aspects like grammar or vocabulary. The researchers tested 25 LLMs and found that while overall scores can be predicted from the model’s internal features, the models don’t treat specific scoring criteria as separate controls. This means that trying to adjust one part of the score, such as grammar, doesn’t work much better than adjusting a different part, reflecting a limitation of current LLM scorers.
What this means in practice
- •For language assessment developers: Improve design of essay scoring tools by recognizing that current LLMs cannot independently control rubric dimensions, guiding better system auditing and enhancement.
- •For educational testing providers: Adjust expectations and quality checks for automated essay scores since current LLM models combine rubric aspects rather than separating them.
Authors
Chungche Huang, Po-Ya Angela Wang, Shu-Kai Hsieh
Abstract
Automated essay scoring (AES) using large language models (LLMs) offers promising use cases for language assessment, yet high-stakes testing requires scores that are both controllable and verifiable, which current LLM-based scorers lack. Given that, we investigate whether LLM-based scorers form distinct internal representations for individual scoring rubrics and whether these representations can be independently controlled. Probing and steering 25 open-weight base LLMs (0.13B-10.31B parameters; 14 families) on human-scored BESTEP English writing tasks, we find that holistic scores are decodable from model activations for the longer tasks (Parts 2 and 3) and only weakly for the shortest (Part 1). Furthermore, scoring rubrics have partly separable directions in representation space for the longer tasks, though not for short responses (Part 1). However, causal steering reveals a critical limitation: applying a matching intervention (e.g., steering "grammar" on a grammar-degraded response) yields only a negligible advantage (about 2% of the effect) over a mismatched intervention (e.g., steering "vocabulary"), and none that is detectable in any of the 10 testable comparisons across reliably measured rubrics. Rather than controlling distinct evaluative dimensions, the rubric directions share a dominant lever whose rubric-specific parts are not selective, which may mirror the halo structure of human ratings. Our contributions are two-fold. First, we show that analytic scoring rubrics designed for human raters do not behave as independent controls in base LLMs, even among the seven models whose scores agree best with human raters ($r \geq 0.6$). Second, we establish a methodology that shows that score agreement and decodability do not imply rubric-level control in LLM-based assessment.