Classroom polls show bias and limited student skill separation
An Audit of Measurement Quality and Answer Bias in a Large Classroom-Poll Corpus
Human-Computer InteractionComputers and Society
Summary
Many classrooms use quick polls to check what students know, but the questions are rarely checked if they measure learning well. The paper looked at over 600 real poll questions answered by thousands of students and found that the questions often don’t clearly separate stronger from weaker students. Also, most students tend to answer 'True' more often, making 'False' answers unexpectedly harder. The authors suggest ways to spot these issues early using data the polling systems already collect.
What this means in practice
- •For online education platforms: Detect and adjust for student answer biases to improve the quality of automated quiz questions before releasing them.
- •For learning management system developers: Use item-level reliability and bias checks during question authoring to ensure poll questions better separate student skill levels.
Authors
Rohit Sharma, Pavani Ayinampudi, Aditya B. M. V., Jinal Gupta, Prakash Hegade, Sakshi Sharma, Meenakshi V, SRS Iyengar
Abstract
Real-time classroom polls are widely used and increasingly generated with automated assistance, yet the questions themselves are rarely evaluated as measurements. We audit a large corpus of authentic classroom polls, 604 items across 47 sessions answered 340,668 times by 2,807 learners, as a measurement instrument. For the 539 items whose correct answer could be established and verified from the lecture transcript, we place every item and every student on a common scale using item response theory and analyse the answer structure of the True/False items. Two findings emerge. First, the polls form a coherent but easy scale of moderate precision (marginal reliability about 0.60), on which roughly a quarter of items barely separate stronger from weaker students. Second, students show a robust tendency to answer True, present at the individual level (77% of students lean True), which meets a milder tendency for items to be keyed False; as a result answer direction predicts difficulty, False-keyed items being about thirteen points harder, and the effect survives controls for item content and for selective answering. Both findings rest on signals a polling system already records, so the same checks can be run as items are generated, before they reach students.