learning analytics teams

Abstract: In K-12 mathematics tutoring, student-tutor dialogue provides rich evidence of learners' problem-solving processes and sources of difficulty. Learning analytics research increasingly relies on large language models (LLMs) to extract such information from dialogue for a variety of downstream tasks, including knowledge tracing, behavioral modeling, and diagnosis of student reasoning errors. However, the validity of these model-generated interpretations remains insufficiently understood. In this exploratory study, we examine the validity of LLM classifications of five student failure modes in mathematics tutoring dialogue using an operational diagnostic codebook: uncertainty, misattribution, operator selection, conceptual gap, and procedural slip. Across models, human-LLM agreement was moderate (kappa = .524-.597), while cross-model agreement was substantially higher (kappa = .755-.781; alpha = .769). These findings show that cross-model agreement can create a misleading appearance of correctness, challenging the assumption that consensus among LLMs constitutes evidence of valid learner interpretation. For learning analytics, the implication is clear: scalable labeling is useful only if the inferred constructs are valid, and model consensus cannot substitute for independent evidence of that validity.

Abstract: Learning analytics increasingly represents the relations in co-occurrence data---codes in a window of discourse, participants in a thread, tags on a post---as simplicial complexes and analyses them with persistent homology. The standard approach uses the clique complex. Because its simplices are determined by the pairwise network alone, it cannot distinguish "three elements co-occurred together" from "each of the three pairs co-occurred separately". The Dowker complex, in contrast, takes as simplices the sets of entities actually observed to co-occur. We compare analyses based on these two complexes. First, we show that, on the same 1-skeleton, the Dowker complex is a subcomplex of the clique complex, the map on first homology induced by the inclusion is surjective, and its kernel is generated by phantom triangles that never co-occurred; that is, the clique construction can only erase holes. We then test this on real data. On four Stack Exchange data sets, 46--91% of clique triangles are phantom and 522--1,970 holes are erased. On the example data of the learning analytics R packages tna/Nestimate, the first Betti number $β_1$ of the clique complex is 0 at every threshold, whereas the Dowker complex detects holes. Against a degree-preserving null model, the Dowker complex departs strongly in five of the six data sets. Moreover, this difference does not appear in the fixed-threshold analyses used in practice. We conclude that the construction should follow the data type and that, for observed groups, the Dowker complex is the appropriate choice; it fills a gap in current tooling. Code is available at https://github.com/igu-lab/beyond-the-clique.

Abstract: Real-time classroom polling is now routine, yet the data it produces is usually read narrowly, as a correctness score or a headcount. Such readings say little about what a poll is doing within a lecture or how it shapes engagement. This is particularly relevant for short-response formats such as True/False, where the same question format can be used to test recall, check comprehension, or direct students' attention to a deliberately misleading statement. This study asks whether a poll's answer and instructional function can be determined by reading it against its lecture transcript, what cognitive levels of Bloom's taxonomy and instructional-function clusters the corpus contains, and how student engagement relates to answering correctly. We analyse a naturalistic corpus of 47 live sessions over 39 days, comprising 604 poll questions and 340,668 responses from 2,807 learners, most items True/False, read against time-aligned lecture transcripts and attendance. Reading each poll in context proves essential: the answer to 89% of polls is locatable in the lecture, and a recurring attention-checking device is visible only through context. Questioning is overwhelmingly lower-order and falls into seven instructional functions, and a poll's response follows its function rather than its wording. Engagement is broad but concentrated, and the class majority answers correctly 88.5% of the time, though a small set of high-consensus yet incorrect answers cannot be detected by agreement alone. An independent survey of 579 students agrees on what the polls are and on their participation, but reveals a gap between perception and reality: students cannot judge their own correctness, and the polls they find hardest are not those they answer worst.

Abstract: Cognitive diagnosis infers students' concept mastery from response logs. However, students' responses are not determined by mastery alone: non-cognitive factors such as emotion, engagement, and fatigue can also affect performance. Affective cognitive diagnosis therefore extends conventional cognitive diagnosis by incorporating affective states. Existing methods often assume that the cognitive diagnosis backbone has already explained ability, item, and concept effects, so the remaining errors can be attributed mainly to affect. We argue that this assumption can be insufficient in real educational data: item calibration bias, systematic concept bias, personalized student-concept deviations, and latent student-item matching can form stable cognitive residuals. Without an explicit modeling pathway, these residuals may leak into affective representations, producing affect contamination. To address this problem, we propose an ability-residual decoupled framework for affective cognitive diagnosis. The model first captures unmodeled cognitive residuals through student, item, concept, student-concept, and low-rank student-item components, and then uses an affective module to modulate guess/slip effects. A Q-matrix-constrained concept residual attention mechanism adaptively aggregates only item-relevant concept residuals. Experiments on ASSIST2017, ASSIST2012, ASSIST2009, and Junyi with six cognitive diagnosis backbones show response-prediction gains across the reported comparisons and generally improved affect alignment when affect labels are available. Ablation studies, leakage probes, principal component analysis visualization, long-tail analysis, and case studies further indicate that ability residuals absorb stable cognitive bias, reduce cognitive contamination in the affective branch, and enhance the robustness and predictive accuracy of cognitive diagnosis models.