Re-Evaluating AI Chatbot Effectiveness: A Systematic Review of Dialogue-Based CALL Research

Chatbots have established themselves as one of the main foci of interest of technology-enhanced language learning research, owing to their ability to partake in natural language exchanges through both the spoken and the written medium. Evidence of their learning benefits both within and outside of the classroom has been examined in a large body of research featuring scoping reviews and meta-analyses, which have also highlighted their potential drawbacks and technical fallacies. The present contribution shifts the focus away from the capabilities and limitations of the technology to address how chatbots’ effectiveness, or lack thereof, is pedagogically framed and experimentally operationalised within dialogue-based computer-assisted language learning (CALL). Thirty-seven empirical studies were sampled to investigate four core dimensions underlying chatbots’ effectiveness for English language practice: learners’ profiles, theoretical frameworks, teaching settings and learning outcomes. Results highlight a limited conceptualisation of chatbots’ effectiveness in L2 English teaching, tied to an under-theorisation of their instructional affordances and a narrow definition of learning trajectories, interaction types and measurable outcomes. These aspects appear to hold true independent of the chatbot architecture employed, whether based on large language models (LLMs) or not, suggesting a misalignment between technology implementation and pedagogical grounding, where the interactionist foundations that originally informed dialogue-based CALL are still insufficiently featured in evaluations of chatbot-mediated language learning.