Turning 500+ Students into Teachers: A Semester-Long Study of an AI Teachable Agent in an Undergraduate Algorithms Course

Abstract

Large language model (LLM) tools can provide students with rapid solutions but may reduce opportunities for productive struggle and explanation generation that support conceptual learning. Learning-by-teaching (LBT) offers an alternative solution by positioning students as tutors; however, evidence for LLM-based teachable agents remains limited, particularly for longitudinal deployments and large-scale evaluations that connect LBT interactions to conceptual understanding in authentic courses. We present Explique, a platform that integrates an AI teachable agent, Algorithm Apprentice, into an undergraduate algorithms course to operationalise LBT at scale. We report an 11-week field deployment in a real course with 546 students, analysing 3,809 student-agent LBT dialogues alongside quiz and survey data. Students engaged consistently in multi-turn teaching interactions over the semester, although the depth and authenticity of these interactions varied, including instances of direct reuse of externally sourced content. Using generalised linear mixed-effects models, we find that explanation-oriented dialogue behaviours (e.g., elaboration and showing reasoning) are associated with fewer quiz attempts (i.e., fewer incorrect submissions), whereas external-content reuse is associated with slightly more repeated attempts. Compared to a baseline reading activity, the LBT condition corresponds to a modest reduction in expected quiz attempts, although this comparison is confounded by substantial differences in time-on-task. Overall, these results provide longitudinal, large-scale evidence on LLM-based teachable agents in an authentic computer science course and inform the design and practice of systems that aim to support sustained, effortful and scalable LBT interactions.