Alice dataset advances German automatic short answer scoring accuracy
Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring
Computation and Language
Summary
Scoring short student answers automatically is hard because existing tests mostly check if answers match expected solutions, not if students understand important concepts or skills behind them. The authors created Alice, a large German dataset that scores answers based on detailed rubrics covering knowledge and skills, not just correct answers. They tested different language models and found that models struggle especially with scoring concept mastery and skills unless given extra rubric details. This work helps build better tools to assess student learning more deeply.
What this means in practice
- •For educational software developers: Create automated grading tools that evaluate student answers for mastery of concepts and skills in German language educational platforms.
- •For language model engineers: Improve language models to better understand and score complex student answers by integrating rubric-based criteria during development.
Authors
Zhifan Sun, Sebastian Gombert, Jannik Lossjew, Tobias Wyrwich, Berrit Katharina Czinczel, David Bednorz, Marcus Kubsch, Knut Neumann, Hendrik Drachsler
Abstract
Automatic Short Answer Scoring (ASAS) is central to NLP for Education. However, openly available benchmarks remain scarce, and existing datasets largely address how well students answer a question directly rather than how well they master underlying concepts (knowledge elements) such as thermal energy or epistemic activities (skills) such as reasoning or claim. To address this gap, we introduce Alice, a large-scale, rubric-based German ASAS dataset that is pedagogically aligned and comprises three subtasks: (i) learning performance (Alice-LP), (ii) knowledge elements (Alice-KE), and (iii) skills (Alice-SK). We further formulate rubric-based ASAS as a rubric-retrieval task and benchmark the dataset with a range of language models, from encoder-only models to lightweight LLMs. We also benchmark the dataset with zero-shot prompting via LLMs and a standard classification baseline. The experiments show that LLMs, in particular, struggle to score knowledge elements and skills in the zero-shot setting. They also indicate that rubric text is often useful, especially for Alice-KE and Alice-SK, while on Alice-LP gains over sample-solution-focused inputs are more modest and vary by model and input format.