corporate training teams

Abstract: Durable skills, such as collaboration, creativity and critical thinking, are instrumental to success in the modern workforce. Yet, measuring these skills remains a persistent challenge. Moreover, because what is not measured is often not taught, these skills are often overlooked in mainstream educational curricula. Designing effective assessments for these skills necessitates balancing two often-conflicting requirements: ecological validity and psychometric rigor. On the one hand, the assessment environment should emulate natural real-world human interaction between humans. On the other hand, it should be scalable, controllable and reproducible. Here we argue that LLMs can be used to better capture both of these aims. Concretely, we develop a framework where the subject converses with AI teammates in a way that resembles human-human interaction for authenticity, while also offering the psychometric control required for informative and robust assessment. Importantly, the AI participants not only act as teammates but also, in an "Executive LLM" setup, steer the conversation towards eliciting a high density of observable evidence for skill proficiency. We complement this with an AI evaluator that can be used to measure skill proficiency in such interactions. We evaluate our assessment protocol based on transcripts of interactions of human participants with our AI framework, for multiple durable skills. For the skill of creativity, we further demonstrate the efficacy of an autorater for evaluating complex tasks performed by real students. Our analysis shows that the use of the Executive LLM significantly increases elicited evidence and that LLM-automated scoring of conversations largely agrees with that of expert annotators. This research demonstrates the utility of orchestrated LLMs approaches for measuring complex social and cognitive constructs in a scalable and controllable manner.

Abstract: As AI and data-driven systems pervade practice, there is an imperative for instructors to embed societal impact and ethics content into computing courses. In response, we present the Human-Centered Education for Learning in Information and eXplainable Computing (HELIX) framework for information science programs, organized around three iterative pillars - knowledge building, decision-making, and empowerment - with concrete actions for instructors and students. We applied the framework in a graduate, introductory programming course using readings, algorithmic design activities, and scenario-based reflections. We present a pilot implementation of this framework to examine changes in students' (n=22) knowledge acquisition, decision-making processes, and self-reflection regarding human-centered perspectives in data science. We release an anonymized materials kit (survey, assignments, analysis code) to support adoption. We discuss design tensions (workload, assessment, relevance to diverse information science learners) and provide guidelines for integrating human-centered content without overwhelming technical outcomes. Findings suggest that the HELIX Framework is feasible in information science contexts and future work should use comparative survey assessment to strengthen causal inferences.