assignment

designing an educational chatbot a case study of cikguaibot

Information

  • Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

  • Active Journals
  • Find a Journal
  • Proceedings Series
  • For Authors
  • For Reviewers
  • For Editors
  • For Librarians
  • For Publishers
  • For Societies
  • For Conference Organizers
  • Open Access Policy
  • Institutional Open Access Program
  • Special Issues Guidelines
  • Editorial Process
  • Research and Publication Ethics
  • Article Processing Charges
  • Testimonials
  • Preprints.org
  • SciProfiles
  • Encyclopedia

sustainability-logo

Article Menu

designing an educational chatbot a case study of cikguaibot

  • Subscribe SciFeed
  • Recommended Articles
  • Google Scholar
  • on Google Scholar
  • Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Educational design principles of using ai chatbot that supports self-regulated learning in education: goal setting, feedback, and personalization.

designing an educational chatbot a case study of cikguaibot

1. Introduction

2. theoretical framework, 2.1. review of zimmerman’s multi-level self-regulated learning framework, 2.2. definition and background of jol, 3. educational principles that guide integration of chatbots, 3.1. define chatbots and describe their potential use in educational settings, 3.2. goal setting and prompting, 3.3. feedback and self-assessment mechanism, 3.4. facilitating self-regulation: personalization and adaptation, 4. limitations, 5. concluding remarks, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

  • Pérez, J.Q.; Daradoumis, T.; Puig, J.M.M. Rediscovering the use of chatbots in education: A systematic literature review. Comput. Appl. Eng. Educ. 2020 , 28 , 1549–1565. [ Google Scholar ] [ CrossRef ]
  • Smutny, P.; Schreiberova, P. Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Comput. Educ. 2020 , 151 , 103862. [ Google Scholar ] [ CrossRef ]
  • Kuhail, M.A.; Alturki, N.; Alramlawi, S.; Alhejori, K. Interacting with educational chatbots: A systematic review. Educ. Inf. Technol. 2023 , 28 , 973–1018. [ Google Scholar ] [ CrossRef ]
  • Okonkwo, C.W.; Ade-Ibijola, A. Chatbots applications in education: A systematic review. Comput. Educ. Artif. Intell. 2021 , 2 , 100033. [ Google Scholar ] [ CrossRef ]
  • Koriat, A. Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning. J. Exp. Psychol. Gen. 1997 , 126 , 349–370. [ Google Scholar ] [ CrossRef ]
  • Son, L.K.; Metcalfe, J. Judgments of learning: Evidence for a two-stage process. Mem. Cogn. 2005 , 33 , 1116–1129. [ Google Scholar ] [ CrossRef ] [ PubMed ]
  • Panadero, E. A Review of Self-regulated Learning: Six Models and Four Directions for Research. Front. Psychol. 2017 , 8 , 422. [ Google Scholar ] [ CrossRef ] [ PubMed ]
  • Zimmerman, B.J. Attaining Self-Regulation. In Handbook of Self-Regulation ; Elsevier: Amsterdam, The Netherlands, 2000; pp. 13–39. [ Google Scholar ] [ CrossRef ]
  • Baars, M.; Wijnia, L.; de Bruin, A.; Paas, F. The Relation Between Students’ Effort and Monitoring Judgments During Learning: A Meta-analysis. Educ. Psychol. Rev. 2020 , 32 , 979–1002. [ Google Scholar ] [ CrossRef ]
  • Leonesio, R.J.; Nelson, T.O. Do different metamemory judgments tap the same underlying aspects of memory? J. Exp. Psychol. Learn. Mem. Cogn. 1990 , 16 , 464–470. [ Google Scholar ] [ CrossRef ]
  • Double, K.S.; Birney, D.P.; Walker, S.A. A meta-analysis and systematic review of reactivity to judgements of learning. Memory 2018 , 26 , 741–750. [ Google Scholar ] [ CrossRef ]
  • Janes, J.L.; Rivers, M.L.; Dunlosky, J. The influence of making judgments of learning on memory performance: Positive, negative, or both? Psychon. Bull. Rev. 2018 , 25 , 2356–2364. [ Google Scholar ] [ CrossRef ]
  • Hamzah, M.I.; Hamzah, H.; Zulkifli, H. Systematic Literature Review on the Elements of Metacognition-Based Higher Order Thinking Skills (HOTS) Teaching and Learning Modules. Sustainability 2022 , 14 , 813. [ Google Scholar ] [ CrossRef ]
  • Veenman, M.V.J.; Van Hout-Wolters, B.H.A.M.; Afflerbach, P. Metacognition and learning: Conceptual and methodological considerations. Metacognition Learn. 2006 , 1 , 3–14. [ Google Scholar ] [ CrossRef ]
  • Nelson, T.; Narens, L. Why investigate metacognition. In Metacognition: Knowing about Knowing ; MIT Press: Cambridge, MA, USA, 1994. [ Google Scholar ] [ CrossRef ]
  • Tuysuzoglu, B.B.; Greene, J.A. An investigation of the role of contingent metacognitive behavior in self-regulated learning. Metacognition Learn. 2015 , 10 , 77–98. [ Google Scholar ] [ CrossRef ]
  • Bandura, A. Social Cognitive Theory: An Agentic Perspective. Asian J. Soc. Psychol. 1999 , 2 , 21–41. [ Google Scholar ] [ CrossRef ]
  • Bem, D.J. Self-Perception Theory. In Advances in Experimental Social Psychology ; Berkowitz, L., Ed.; Academic Press: Cambridge, MA, USA, 1972; Volume 6, pp. 1–62. [ Google Scholar ] [ CrossRef ]
  • Abu Shawar, B.; Atwell, E. Different measurements metrics to evaluate a chatbot system. In Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, Rochester, NY, USA, 26 April 2007; pp. 89–96. [ Google Scholar ] [ CrossRef ]
  • Turing, A.M. Computing machinery and intelligence. Mind 1950 , 59 , 433–460. [ Google Scholar ] [ CrossRef ]
  • Weizenbaum, J. ELIZA—A computer program for the study of natural language communication between man and machine. Commun. ACM 1966 , 9 , 36–45. [ Google Scholar ] [ CrossRef ]
  • Wallace, R.S. The anatomy of A.L.I.C.E. In Parsing the Turing Test: Philosophical and Methodological Issues in the Quest for the Thinking Computer ; Epstein, R., Roberts, G., Beber, G., Eds.; Springer: Dordrecht, The Netherlands, 2009; pp. 181–210. [ Google Scholar ] [ CrossRef ]
  • Hwang, G.-J.; Chang, C.-Y. A review of opportunities and challenges of chatbots in education. Interact. Learn. Environ. 2021 , 1–14. [ Google Scholar ] [ CrossRef ]
  • Yamada, M.; Goda, Y.; Matsukawa, H.; Hata, K.; Yasunami, S. A Computer-Supported Collaborative Learning Design for Quality Interaction. IEEE MultiMedia 2016 , 23 , 48–59. [ Google Scholar ] [ CrossRef ]
  • Muniasamy, A.; Alasiry, A. Deep Learning: The Impact on Future eLearning. Int. J. Emerg. Technol. Learn. (iJET) 2020 , 15 , 188–199. [ Google Scholar ] [ CrossRef ]
  • Bendig, E.; Erb, B.; Schulze-Thuesing, L.; Baumeister, H. The Next Generation: Chatbots in Clinical Psychology and Psychotherapy to Foster Mental Health—A Scoping Review. Verhaltenstherapie 2022 , 32 , 64–76. [ Google Scholar ] [ CrossRef ]
  • Kennedy, C.M.; Powell, J.; Payne, T.H.; Ainsworth, J.; Boyd, A.; Buchan, I. Active Assistance Technology for Health-Related Behavior Change: An Interdisciplinary Review. J. Med. Internet Res. 2012 , 14 , e80. [ Google Scholar ] [ CrossRef ]
  • Poncette, A.-S.; Rojas, P.-D.; Hofferbert, J.; Sosa, A.V.; Balzer, F.; Braune, K. Hackathons as Stepping Stones in Health Care Innovation: Case Study with Systematic Recommendations. J. Med. Internet Res. 2020 , 22 , e17004. [ Google Scholar ] [ CrossRef ] [ PubMed ]
  • Ferrell, O.C.; Ferrell, L. Technology Challenges and Opportunities Facing Marketing Education. Mark. Educ. Rev. 2020 , 30 , 3–14. [ Google Scholar ] [ CrossRef ]
  • Behera, R.K.; Bala, P.K.; Ray, A. Cognitive Chatbot for Personalised Contextual Customer Service: Behind the Scene and beyond the Hype. Inf. Syst. Front. 2021 , 1–21. [ Google Scholar ] [ CrossRef ]
  • Crolic, C.; Thomaz, F.; Hadi, R.; Stephen, A.T. Blame the Bot: Anthropomorphism and Anger in Customer–Chatbot Interactions. J. Mark. 2022 , 86 , 132–148. [ Google Scholar ] [ CrossRef ]
  • Clarizia, F.; Colace, F.; Lombardi, M.; Pascale, F.; Santaniello, D. Chatbot: An education support system for student. In CSS 2018: Cyberspace Safety and Security ; Castiglione, A., Pop, F., Ficco, M., Palmieri, F., Eds.; Lecture Notes in Computer Science Book Series; Springer International Publishing: Cham, Switzerland, 2018; Volume 11161, pp. 291–302. [ Google Scholar ] [ CrossRef ]
  • Firat, M. What ChatGPT means for universities: Perceptions of scholars and students. J. Appl. Learn. Teach. 2023 , 6 , 57–63. [ Google Scholar ] [ CrossRef ]
  • Kim, H.-S.; Kim, N.Y. Effects of AI chatbots on EFL students’ communication skills. Commun. Ski. 2021 , 21 , 712–734. [ Google Scholar ]
  • Hill, J.; Ford, W.R.; Farreras, I.G. Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Comput. Hum. Behav. 2015 , 49 , 245–250. [ Google Scholar ] [ CrossRef ]
  • Wu, E.H.-K.; Lin, C.-H.; Ou, Y.-Y.; Liu, C.-Z.; Wang, W.-K.; Chao, C.-Y. Advantages and Constraints of a Hybrid Model K-12 E-Learning Assistant Chatbot. IEEE Access 2020 , 8 , 77788–77801. [ Google Scholar ] [ CrossRef ]
  • Brandtzaeg, P.B.; Følstad, A. Why people use chatbots. In INSCI 2017: Internet Science ; Kompatsiaris, I., Cave, J., Satsiou, A., Carle, G., Passani, A., Kontopoulos, E., Diplaris, S., McMillan, D., Eds.; Lecture Notes in Computer Science Book Series; Springer International Publishing: Cham, Switzerland, 2017; Volume 10673, pp. 377–392. [ Google Scholar ] [ CrossRef ]
  • Deng, X.; Yu, Z. A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education. Sustainability 2023 , 15 , 2940. [ Google Scholar ] [ CrossRef ]
  • de Quincey, E.; Briggs, C.; Kyriacou, T.; Waller, R. Student Centred Design of a Learning Analytics System. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, Tempe, AZ, USA, 4 March 2019; pp. 353–362. [ Google Scholar ] [ CrossRef ]
  • Hattie, J. The black box of tertiary assessment: An impending revolution. In Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research ; Ako Aotearoa: Wellington, New Zealand, 2009; pp. 259–275. [ Google Scholar ]
  • Wisniewski, B.; Zierer, K.; Hattie, J. The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research. Front. Psychol. 2020 , 10 , 3087. [ Google Scholar ] [ CrossRef ]
  • Winne, P.H. Cognition and metacognition within self-regulated learning. In Handbook of Self-Regulation of Learning and Performance , 2nd ed.; Routledge: London, UK, 2017. [ Google Scholar ]
  • Serban, I.V.; Sankar, C.; Germain, M.; Zhang, S.; Lin, Z.; Subramanian, S.; Kim, T.; Pieper, M.; Chandar, S.; Ke, N.R.; et al. A deep reinforcement learning chatbot. arXiv 2017 , arXiv:1709.02349. [ Google Scholar ]
  • Shneiderman, B.; Plaisant, C. Designing the User Interface: Strategies for Effective Human-Computer Interaction , 4th ed.; Pearson: Boston, MA, USA; Addison Wesley: Hoboken, NJ, USA, 2004. [ Google Scholar ]
  • Abbasi, S.; Kazi, H. Measuring effectiveness of learning chatbot systems on student’s learning outcome and memory retention. Asian J. Appl. Sci. Eng. 2014 , 3 , 57–66. [ Google Scholar ] [ CrossRef ]
  • Winkler, R.; Soellner, M. Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. Acad. Manag. Proc. 2018 , 2018 , 15903. [ Google Scholar ] [ CrossRef ]
  • Dwivedi, Y.K.; Kshetri, N.; Hughes, L.; Slade, E.L.; Jeyaraj, A.; Kar, A.K.; Baabdullah, A.M.; Koohang, A.; Raghavan, V.; Ahuja, M.; et al. Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Int. J. Inf. Manag. 2023 , 71 , 102642. [ Google Scholar ] [ CrossRef ]
  • Dai, W.; Lin, J.; Jin, F.; Li, T.; Tsai, Y.S.; Gasevic, D.; Chen, G. Can large language models provide feedback to students? A case study on ChatGPT. 2023; preprint . [ Google Scholar ] [ CrossRef ]
  • Lin, M.P.-C.; Chang, D. Enhancing post-secondary writers’ writing skills with a Chatbot: A mixed-method classroom study. J. Educ. Technol. Soc. 2020 , 23 , 78–92. [ Google Scholar ]
  • Zhu, C.; Sun, M.; Luo, J.; Li, T.; Wang, M. How to harness the potential of ChatGPT in education? Knowl. Manag. E-Learn. 2023 , 15 , 133–152. [ Google Scholar ] [ CrossRef ]
  • Prøitz, T.S. Learning outcomes: What are they? Who defines them? When and where are they defined? Educ. Assess. Eval. Account. 2010 , 22 , 119–137. [ Google Scholar ] [ CrossRef ]
  • Burke, J. (Ed.) Outcomes, Learning and the Curriculum: Implications for Nvqs, Gnvqs and Other Qualifications ; Routledge: London, UK, 1995. [ Google Scholar ] [ CrossRef ]
  • Locke, E.A. New Developments in Goal Setting and Task Performance , 1st ed.; Routledge: London, UK, 2013. [ Google Scholar ] [ CrossRef ]
  • Leake, D.B.; Ram, A. Learning, goals, and learning goals: A perspective on goal-driven learning. Artif. Intell. Rev. 1995 , 9 , 387–422. [ Google Scholar ] [ CrossRef ]
  • Greene, J.A.; Azevedo, R. A Theoretical Review of Winne and Hadwin’s Model of Self-Regulated Learning: New Perspectives and Directions. Rev. Educ. Res. 2007 , 77 , 334–372. [ Google Scholar ] [ CrossRef ]
  • Pintrich, P.R. A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students. Educ. Psychol. Rev. 2004 , 16 , 385–407. [ Google Scholar ] [ CrossRef ]
  • Schunk, D.H.; Greene, J.A. (Eds.) Handbook of Self-Regulation of Learning and Performance , 2nd ed.; In Educational Psychology Handbook Series; Routledge: New York, NY, USA; Taylor & Francis Group: Milton Park, UK, 2018. [ Google Scholar ]
  • Chen, C.-H.; Su, C.-Y. Using the BookRoll e-book system to promote self-regulated learning, self-efficacy and academic achievement for university students. J. Educ. Technol. Soc. 2019 , 22 , 33–46. [ Google Scholar ]
  • Michailidis, N.; Kapravelos, E.; Tsiatsos, T. Interaction analysis for supporting students’ self-regulation during blog-based CSCL activities. J. Educ. Technol. Soc. 2018 , 21 , 37–47. [ Google Scholar ]
  • Paans, C.; Molenaar, I.; Segers, E.; Verhoeven, L. Temporal variation in children’s self-regulated hypermedia learning. Comput. Hum. Behav. 2019 , 96 , 246–258. [ Google Scholar ] [ CrossRef ]
  • Morisano, D.; Hirsh, J.B.; Peterson, J.B.; Pihl, R.O.; Shore, B.M. Setting, elaborating, and reflecting on personal goals improves academic performance. J. Appl. Psychol. 2010 , 95 , 255–264. [ Google Scholar ] [ CrossRef ]
  • Krathwohl, D.R. A Revision of Bloom’s Taxonomy: An Overview. Theory Pract. 2002 , 41 , 212–218. [ Google Scholar ] [ CrossRef ]
  • Bouffard, T.; Boisvert, J.; Vezeau, C.; Larouche, C. The impact of goal orientation on self-regulation and performance among college students. Br. J. Educ. Psychol. 1995 , 65 , 317–329. [ Google Scholar ] [ CrossRef ]
  • Javaherbakhsh, M.R. The Impact of Self-Assessment on Iranian EFL Learners’ Writing Skill. Engl. Lang. Teach. 2010 , 3 , 213–218. [ Google Scholar ] [ CrossRef ]
  • Zepeda, C.D.; Richey, J.E.; Ronevich, P.; Nokes-Malach, T.J. Direct instruction of metacognition benefits adolescent science learning, transfer, and motivation: An in vivo study. J. Educ. Psychol. 2015 , 107 , 954–970. [ Google Scholar ] [ CrossRef ]
  • Ndoye, A. Peer/self assessment and student learning. Int. J. Teach. Learn. High. Educ. 2017 , 29 , 255–269. [ Google Scholar ]
  • Schunk, D.H. Goal and Self-Evaluative Influences During Children’s Cognitive Skill Learning. Am. Educ. Res. J. 1996 , 33 , 359–382. [ Google Scholar ] [ CrossRef ]
  • King, A. Enhancing Peer Interaction and Learning in the Classroom Through Reciprocal Questioning. Am. Educ. Res. J. 1990 , 27 , 664–687. [ Google Scholar ] [ CrossRef ]
  • Mason, L.H. Explicit Self-Regulated Strategy Development Versus Reciprocal Questioning: Effects on Expository Reading Comprehension Among Struggling Readers. J. Educ. Psychol. 2004 , 96 , 283–296. [ Google Scholar ] [ CrossRef ]
  • Newman, R.S. Adaptive help seeking: A strategy of self-regulated learning. In Self-Regulation of Learning and Performance: Issues and Educational Applications ; Lawrence Erlbaum Associates, Inc.: Hillsdale, NJ, USA, 1994; pp. 283–301. [ Google Scholar ]
  • Rosenshine, B.; Meister, C. Reciprocal Teaching: A Review of the Research. Rev. Educ. Res. 1994 , 64 , 479–530. [ Google Scholar ] [ CrossRef ]
  • Baleghizadeh, S.; Masoun, A. The Effect of Self-Assessment on EFL Learners’ Self-Efficacy. TESL Can. J. 2014 , 31 , 42. [ Google Scholar ] [ CrossRef ]
  • Moghadam, S.H. What Types of Feedback Enhance the Effectiveness of Self-Explanation in a Simulation-Based Learning Environment? Available online: https://summit.sfu.ca/item/34750 (accessed on 14 July 2023).
  • Vanichvasin, P. Effects of Visual Communication on Memory Enhancement of Thai Undergraduate Students, Kasetsart University. High. Educ. Stud. 2020 , 11 , 34–41. [ Google Scholar ] [ CrossRef ]
  • Schumacher, C.; Ifenthaler, D. Features students really expect from learning analytics. Comput. Hum. Behav. 2018 , 78 , 397–407. [ Google Scholar ] [ CrossRef ]
  • Marzouk, Z.; Rakovic, M.; Liaqat, A.; Vytasek, J.; Samadi, D.; Stewart-Alonso, J.; Ram, I.; Woloshen, S.; Winne, P.H.; Nesbit, J.C. What if learning analytics were based on learning science? Australas. J. Educ. Technol. 2016 , 32 , 1–18. [ Google Scholar ] [ CrossRef ]
  • Akhtar, S.; Warburton, S.; Xu, W. The use of an online learning and teaching system for monitoring computer aided design student participation and predicting student success. Int. J. Technol. Des. Educ. 2015 , 27 , 251–270. [ Google Scholar ] [ CrossRef ]
  • Lo, C.K. What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Educ. Sci. 2023 , 13 , 410. [ Google Scholar ] [ CrossRef ]
  • Baidoo-Anu, D.; Ansah, L.O. Education in the era of generative Artificial Intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. SSRN Electron. J. 2023 , 1–22. [ Google Scholar ] [ CrossRef ]
  • Mogali, S.R. Initial impressions of ChatGPT for anatomy education. Anat. Sci. Educ. 2023 , 1–4. [ Google Scholar ] [ CrossRef ] [ PubMed ]

Click here to enlarge figure

Prompt TypesProcess-BasedOutcome-Based
CognitiveUnderstand
Remember		Create
Apply
MetacognitiveEvaluate
The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

Chang, D.H.; Lin, M.P.-C.; Hajian, S.; Wang, Q.Q. Educational Design Principles of Using AI Chatbot That Supports Self-Regulated Learning in Education: Goal Setting, Feedback, and Personalization. Sustainability 2023 , 15 , 12921. https://doi.org/10.3390/su151712921

Chang DH, Lin MP-C, Hajian S, Wang QQ. Educational Design Principles of Using AI Chatbot That Supports Self-Regulated Learning in Education: Goal Setting, Feedback, and Personalization. Sustainability . 2023; 15(17):12921. https://doi.org/10.3390/su151712921

Chang, Daniel H., Michael Pin-Chuan Lin, Shiva Hajian, and Quincy Q. Wang. 2023. "Educational Design Principles of Using AI Chatbot That Supports Self-Regulated Learning in Education: Goal Setting, Feedback, and Personalization" Sustainability 15, no. 17: 12921. https://doi.org/10.3390/su151712921

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

MDPI

Subscribe to receive issue release notifications and newsletters from MDPI journals

  •  Sign into My Research
  •  Create My Research Account
  • Company Website
  • Our Products
  • About Dissertations
  • Español (España)
  • Support Center

Select language

  • Bahasa Indonesia
  • Português (Brasil)
  • Português (Portugal)

Welcome to My Research!

You may have access to the free features available through My Research. You can save searches, save documents, create alerts and more. Please log in through your library or institution to check if you have access.

Welcome to My Research!

Translate this article into 20 different languages!

If you log in through your library or institution you might have access to this article in multiple languages.

Translate this article into 20 different languages!

Get access to 20+ different citations styles

Styles include MLA, APA, Chicago and many more. This feature may be available for free if you log in through your library or institution.

Get access to 20+ different citations styles

Looking for a PDF of this document?

You may have access to it for free by logging in through your library or institution.

Looking for a PDF of this document?

Want to save this document?

You may have access to different export options including Google Drive and Microsoft OneDrive and citation management tools like RefWorks and EasyBib. Try logging in through your library or institution to get access to these tools.

Want to save this document?

  • Document 1 of 1
  • More like this
  • Scholarly Journal

Educational Design Principles of Using AI Chatbot That Supports Self-Regulated Learning in Education: Goal Setting, Feedback, and Personalization

No items selected.

Please select one or more items.

1. Introduction

Educational chatbots, also called conversational agents, hold immense potential in delivering personalized and interactive learning experiences to students [1,2]. However, the advent of ChatGPT or generative AI poses a substantial challenge to the role of educators, as it gives rise to concerns that students may exploit generative AI tools to obtain academic recognition without actively engaging in the learning process. In light of this transformative development, it is observable that AI represents a contemporary trend in education, and it will be used by learners inevitably. Rather than attempting to suppress using AI in education, educators should proactively seek and explore ways to adapt its presence. This adaptation can be effectively achieved by establishing fruitful collaborations between educators, instructional designers, and researchers in the AI field. Such partnerships should strive to explore the integration of pedagogical principles within AI platforms, ensuring that students not only derive benefits from AI but also acquire the essential skills mandated by the educational curriculum. Consequently, it becomes crucial for chatbot designers and educators to collaborate closely, considering key pedagogical principles such as goal setting, self-assessment, and personalization at various stages of learning [3,4]. These principles should guide the design process, guaranteeing that the chatbot effectively supports the student learning experience.

In this paper, drawing from Barry Zimmerman’s Self-Regulated Learning (SRL) framework, we propose several key pedagogical principles that can be considered when teachers decide to integrate AI chatbots in classrooms in order to foster SRL. While composing this paper, it is evident that the majority of research on generative AI tools (like ChatGPT) mainly focuses on their wide-ranging applications and potential drawbacks in education. However, there has been a notable shortage of studies that actively engage teachers and instructional designers to determine the most effective ways to incorporate these AI tools in classroom settings. In one of our pedagogical principles, we will specifically draw on Judgement of Learning (JOL), which refers to assessing and evaluating student understanding and progress [5,6], and explore how JOL can be integrated into AI-based pedagogy and instructional design that fosters students’ SRL. By integrating Zimmerman’s SRL framework with JOL, we hope to address the major cognitive, metacognitive, and socio-educational concerns contributing to the enhancement and personalization of AI in teaching and learning.

2. Theoretical Framework

Let us conceptualize a learning scenario on writing and learning. A student accesses their institution’s learning management system (LMS) and selects the course titled “ENGL 100—Introduction to Literature”, a foundational writing course under the Department of English. Upon navigating to an assignment, the student delves into its details and reads the assignment instructions. After a brief review, the student copies the assignment’s instructions. In a separate browser tab, the student opens up ChatGPT and decides to engage with it. The student then pastes the assignment instructions, prompting ChatGPT with, “Plan my essay based on the provided assignment instructions, [copied assignment instructions]”.

In response, ChatGPT outlines a structured plan, beginning with the crafting of an introduction. However, the student is puzzled about the nature and structure of an introduction, so the student inquires and re-prompts again, “Could you provide an example instruction for the assignment?” ChatGPT then offers a sample. After studying the example, the student clicks a word processing software on their computer and commences the writing process. Upon completing the introduction, the student seeks feedback from ChatGPT, asking, “Could you assess and evaluate the quality of my introduction?” ChatGPT provides its evaluation. Throughout the writing process, the student frequently consults ChatGPT for assistance on various elements, such as topic sentences, examples, and argumentation, refining their work until the student is satisfied with the work they produce for the ENGL 100 assignment.

This scenario depicts a perfect and ideal SRL cycle executed by the student, encompassing goal-setting, standard reproduction, iterative engagement with ChatGPT, and solicitation of evaluative feedback. However, in real-world educational contexts, students might not recognize this cycle. They might perceive ChatGPT merely as a problem-solving AI chatbot, which can help them with the assignment. On the side of instructors, instructors are not fully aware of how AI tools can be integrated as part of their pedagogy, yet they are afraid that students will use this AI chatbot unethically for learning.

In our position, we argue that generative AI tools, like ChatGPT, have the potential to facilitate SRL when instructors are aware of the SRL process from which students can benefit. To harness the potential of generative AI tools, educators must be cognizant of their capabilities, functions, and pedagogical values. To this end, we employ Zimmerman’s multi-level SRL framework, which will be elaborated upon in the subsequent section.

2.1. Review of Zimmerman’s Multi-Level Self-Regulated Learning Framework

Zimmerman’s multi-level SRL framework [7,8] encompasses four distinct levels: observation, emulation, self-control, and self-regulation (see Figure 1). Each level represents a progressive stage in the development of SRL skills. This framework guides us to explore how a chatbot can facilitate SRL at each stage of Zimmerman’s framework. For example, when students use AI chatbots for their learning, they treat the chatbots as a resource. They enter questions or commands into the AI chatbots, hoping to seek clarifications or information from the chatbots for the task at hand. We assume that this type of utilization of AI chatbots elicits students’ self-regulation. We propose that Zimmerman’s multi-level SRL framework helps to interpret the SRL processes undertaken by students.

Specifically, the observation level denotes a stage where students possess prior knowledge of how conversations occur in a real-life context and their general goal for the learning task. During this phase, students may set their goals, or primarily observe and learn from others who prompt the chatbot, gaining insights into the expected outcomes and interactions. Moving onto the emulation level, students demonstrate their comprehension of the task requirements by independently prompting the chatbot using their own words or similar phrases they have observed or recommended by others. At this stage, students strive to replicate successful interactions they have witnessed, applying their understanding of the task to engage with the chatbot. At this stage, they may also use their goals as the prompts being fed into a chatbot, or they can use the prompts they observe from others. The self-control level, on the other hand, represents a critical juncture where students face decisions about their learning. Such decisions can be ethical conduct and academic integrity decisions, or further re-engagement (re-prompting the chatbot). Specifically, once the chatbot generates a response, students must choose between potentially resorting to taking the chatbot’s responses verbatim for the assignments (academic integrity and ethical conduct) and modifying their approach, such as re-prompting, or sorting out other strategies. This phase provides an opportunity for the chatbot to contribute by offering evaluations and feedback on students’ work, guiding them to determine whether their output meets the required standards or if further revisions are necessary. In sum, at this self-control stage, it can be considered as a two-way interaction between the chatbot and students.

As students march into the self-regulation level when they use the chatbot, they begin to recognize the potential benefits of the chatbot as a useful, efficient, and valuable learning tool to assist their learning. Regarding the self-regulation level, students may seek an evaluation of their revised paragraph generated by the chatbot. Moreover, they might request the chatbot to provide their learning analytics report. Fine-grained student data can be visualized as learning analytics in the chatbot or receive recommendations for further learning improvement. This stage exemplifies the students’ growing understanding of how the chatbot can facilitate their learning process, guiding them toward achieving specific objectives and refining their SRL skills. Zimmerman’s multi-level SRL framework provides a comprehensive perspective on the gradual development of increasing SRL abilities. It illustrates how students proceed from observing and emulating others, exercising self-control, and ultimately achieving self-regulation by harnessing the chatbot’s capabilities as a supportive learning resource.

2.2. Definition and Background of JOL

In Zimmerman’s self-control and self-regulation phases of SRL, students have to engage in some levels of judgement about the chatbot’s output, so they can decide what their next actions are. Such judgement is known as self-assessment, and self-assessment is grounded in Judgement of Learning (JOL), a concept dominant in educational psychology.

JOL is a psychological and educational concept that refers to an individual’s evaluation of their learning [6]. It reflects the extent to which an individual believes they have learned or retained new information, which can impact their motivation and behavior during the learning process [5]. Several studies have indicated that various factors could impact an individual’s JOL, including the difficulty of the material, the individual’s pre-existing knowledge and skills, and the effectiveness of the learning strategy used [5,6]. There is empirical evidence showing that people with a higher JOL tend to be more motivated to learn and more likely to engage in SRL activities, while those with a lower JOL may be less motivated and avoid difficult learning tasks [9,10]. JOL can also serve as a feedback mechanism for learners by allowing them to identify areas where they need to focus more effort and adjust their learning strategies accordingly [11,12]. Additionally, JOL can influence an individual’s confidence, which in turn can affect their overall approach to learning [11].

One of the most influential theories of JOL is the cue-utilization approach, which proposes that individuals use various cues, or indicators, to assess their learning [5]. These cues can include things like how difficult the material was to learn, how much time was spent studying, and how well the material was understood. According to Koriat [5], individuals are more likely to have higher JOL if they encounter more favorable cues while learning (e.g., domain-specific knowledge), and more likely to have a low JOL if they encounter less favorable cues (e.g., feelings of unfamiliarity or difficulty). Another important outcome of JOL is metacognitive awareness, which emphasizes the role of metacognitive processes, or higher-order thinking skills, in the learning process. Research [13,14] shows that individuals use metacognitive strategies, such as planning, monitoring, and evaluating, to guide their learning and assess their progress. As a result, individuals with higher JOL are more likely to use effective metacognitive strategies and be more successful learners. In certain conditions, students recognize their lack of understanding of specific concepts, a phenomenon referred to as “negative JOL” [15], which may result in the improvement of previously adopted learning skills and strategies. Suppose the student does not change their strategy use following such judgement. In that case, the student’s metacognitive behavior is called “static”, implying that they are aware of their knowledge deficit but are resistant to change [16]. Various models of JOL have been proposed. For example, the social cognitive model [17] emphasizes the influence of social and environmental factors on learning, and the self-perception model suggests that individuals’ JOL is influenced by their perceptions of their abilities and self-worth [18].

Taken together, incorporating Zimmerman’s SRL theoretical framework and JOL into the existing capacity of AI in Education has significant potential for improving students’ SRL. Currently, AI technology operates in a unidirectional manner, where users (or students) prompt the generative AI tool to fulfill its intended function and purposes (in the following section, we also call it “goal setting”), as what we have shown above with respect to the emulation and the self-control stages. However, in education, it is crucial to emphasize the importance of bidirectional interaction (from user to AI and AI to user). Enabling AI to initiate personalized learning feedback (i.e., learning analytics, which we will elaborate in the Section 3.4) to users can create meaningful and educational interactions. In the sections below, we propose several educational principles that can guide the integration of chatbots into various aspects of educational practices.

3. Educational Principles That Guide Integration of Chatbots

3.1. Define Chatbots and Describe Their Potential Use in Educational Settings

The term “chatbot” refers to computer programs that communicate with users using natural language [19]. The history of chatbots can be extended back to the early 1950s [20]. In particular, ELIZA [21] and A.L.I.C.E. [22] were well-known early chatbot systems simulating real human communication. Chatbots are technological innovations that may efficiently supplement services delivered to humans. In addition to educational chatbots [23,24] and applying deep learning algorithms in learning management systems [25], chatbots have been used as a tool for many purposes and have a wide range of industrial applications, such as medical education [26,27], counseling consultations [28], marketing education [29], and telecommunications support and in financial industries [30,31].

In particular, research has been conducted to investigate the methods and impacts of chatbot implementation in education in recent years [25,32,33]. Chatbots’ interactive learning feature and their flexibility in terms of time and location have made their usage more appealing and gained popularity in the field of education [23]. Several studies have shown that utilizing chatbots in educational settings may provide students with a positive learning experience, as human-to-chatbot interaction allows real-time engagement [34], improves students’ communication skills [35], and improves students’ efficiency of learning [36].

The growing need for AI technology has opened a new avenue for constructing chatbots when combined with natural language processing capabilities and machine learning techniques [37]. Smutny and Schreiberova’s study [2] showed that chatbots have the potential to become smart teaching assistants in the future, as they might be capable of supplementing in-class instructions alongside instructors. In the case of ChatGPT, some students might have used it as personal assistants, regardless of its underlying ethical conduct in academia. However, we would like to argue that utilizing generative AI chatbots, like ChatGPT, can be a platform for students to become self-regulated under the conditions that they are taught about the context of appropriate use, such as when, where, and how they should use the AI chatbot system for learning. In addition, according to a meta-analysis conducted by Deng and Yu [38], chatbots can potentially have a medium-to-high effect on achievement or learning outcomes. Therefore, integrating AI chatbots into classrooms has now been a question of how educators should do it appropriately to foster learning rather than how educators should suppress it so students will observe the boundary of ethical conduct.

Conventional teaching approaches, such as giving students feedback, encouraging students, or customizing course material to student groups, are still dominant pedagogical practices. Suppose we can take these conventional approaches into account while integrating AI into pedagogy. In that case, we believe that computers and other digital gadgets can bring up far-reaching possibilities that have yet to be completely realized. For example, incorporating process data in student learning may offer students some opportunities to monitor their understanding of materials as well as additional opportunities for formative feedback, self-reflection, and competence development [39]. Hattie [40] has argued that the effect of feedback has a median effect size of d = 0.75 in terms of achievement. On the other hand, Wisniewski et al. [41] have shown that feedback can produce an effect size of d = 0.99 for highly informative feedback on student achievement. Such feedback may foster an SRL process and strong metacognitive monitoring and control [8,15,42]. With these pieces of evidence, we can propose that AI that model teachers’ scaffolding and feedback mechanism after students prompt the AI will support SRL activities.

As stated earlier, under the unidirectional condition (student-to-AI), it has been unclear what instructional and pedagogical functions of chatbots can serve to produce learning effects. In particular, it is unclear what teaching and learning implications are when students use a chatbot to learn. We, therefore, propose an educational framework for integrating an AI educational chatbot based on learning science—Zimmerman’s SRL framework along with JOL.

To our best knowledge, the design of chatbots has focused greatly on the backend design [43], user interface [44], and improving learning [36,45,46]. For example, Winkler and Söllner [46] reviewed the application of chatbots in improving student learning outcomes and suggested that chatbots could support individuals’ development of procedural knowledge and competency skills such as information searching, data collection, decision making, and analytical thinking.

Specifically for learning improvement, since the rise of Open AI’s ChatGPT, there have been several emerging calls for examining how ChatGPT can be integrated pedagogically to support the SRL process. As Dwivedi et al. [47] writes, “Applications like ChatGPT can be used either as a companion or tutor, [or] to support … self-regulated learning” [47] (p. 9). A recent case study also found that ChatGPT gave feedback to student assignments is comparable to that of a human instructor [48]. Lin and Chang’s study [49] and Lin’s doctoral dissertation have also provided a clear bulletin for designing and implementing chatbots for educational purposes and documented several interaction pathways leading to effective peer reviewing activities and writing achievement [49]. Similarly, Zhu et al. [50] argued that “self-regulated learning has been widely promoted in educational settings, the provision of personalized support to sustain self-regulated learning is crucial but inadequately accomplished” (p. 146). Therefore, we are addressing the emerging need to integrate chatbots in education and how chatbots can be developed or used to support learners’ SRL activities. This will be the reason why the fundamental educational principles of pedagogical AI chatbots need to be established. To do so, we have identified several instructional dimensions that we argue should be featured in the design of educational chatbots to facilitate effective learning for students or at least to supplement classroom instructions. These instructional dimensions include (1) goal setting, (2) feedback and self-assessment, and (3) personalization and adaptation.

3.2. Goal Setting and Prompting

Goals and motivation are two highly correlated constructs in education. These two instructional dimensions can guide the design of educational chatbots. In the field of education, the three terms, learning goals, learning objectives, and learning outcomes, have been used interchangeably, though with some conceptual differences [51]. Prøitz [51] mentioned: “the two terms [learning outcomes and learning objectives] are often intertwined and interconnected in the literature makes it difficult to distinguish between them” (p. 122). In the context of SRL and AI chatbots, we argue that the three are inherently similar to some extent. It is because, according to Burke [52] and Prøitz [51], these teacher-written statements contain learning orientation and purpose orientation that manifest their expectations from students. Therefore, these orientations can serve as process-oriented or result-oriented goals that guide learners’ strategies and SRL activities.

In goal-setting theory, learning goals (objectives or outcomes) that are process-oriented, specific, challenging, and achievable can motivate students and serve SRL functions. For instance, Locke and Latham [53] explained that goals may help shape students’ strategies to tackle a learning task, monitor their progress in a studying session, and increase engagement and motivation. Let us take a scenario. Imagine that a student needs to write a report. This result-oriented goal can give rise to two process-based sub-goals: first, they want to synthesize information A, B, and C during a writing session. Secondly, they want to generate an argument. In order to synthesize information, the student may need to apply some strategies. The student’s synthesis goal can drive the student to use some process-oriented writing strategies, such as classifying, listing, or comparing and contrasting. To generate an argument, the student may need to find out what is missing in the synthesized information or what is common among the syntheses. Thus, this example demonstrates that goals articulate two dimensions of learning: the focus of attention and resources needed to achieve the result. As Leake and Ram [54] argued, “a goal-driven learner determines what to learn by reasoning about the information it needs, and determines how to learn by reasoning about the relative merits of alternative learning strategies in the current circumstances” (p. 389).

SRL also consists of learners exercising their metacognitive control and metacognitive monitoring. These two processes are guided by pre-determined result-oriented outcomes: objectives or goals [8,42,55,56,57]. SRL researchers generally agree that goals can trigger several SRL events and metacognitive activities that should be investigated as they occur during learning and problem-solving activities [55,58,59]. Moreover, Paans et al.’s study [60] argues that learner-initiated SRL activities occurring at the micro-level and macro-level can be developed and occur simultaneously, including goal setting or knowledge acquisition. It implies that, in certain pedagogical tasks or problem-solving environments, such as working with chatbots, students need to identify goals by prompting the AI chatbot in a learning session corresponding to the tasks.

Additionally, goals can function as benchmarks by which learners assess the efficacy of their learning endeavors. When students possess the capacity to monitor their progress toward these goals, they are more likely to sustain their motivation and active involvement in the learning process [61]. Within the context of AI chatbot interaction, consider a scenario where a student instructs a chatbot to execute certain actions, such as synthesizing a given set of information. Subsequently, the chatbot provides the requested synthesis, allowing students to evaluate its conformity with their expectations and the learning context. Within Zimmerman’s framework of Self-Regulated Learning, this process aligns with the stages of emulation and self-control. Once a student prompts the chatbot for a response, they continuously monitor and self-assess its quality, subsequently re-prompting the chatbot for further actions. This bidirectional interaction transpires within the stages of simulation and self-control, as students actively participate in a cycle of prompts, monitoring and adjustments, and subsequent re-prompts, persisting until they attain a satisfactory outcome. Yet we have to acknowledge that the interaction assumes student autonomy, in which students keep prompting the chatbot and relying on the chatbot’s output. A more sophisticated way of student–chatbot interaction is bidirectional, where a chatbot is capable of reverse prompting, a concept which we will dive into deeper in our next section.

We believe it is crucial to teach students how to effectively prompt a generative AI chatbot. As we mentioned earlier, prompts are the goals that students set for the AI chatbot, but often students just want the tool’s output without engaging in the actual process. To better understand this, we can break prompts down into two types: cognitive prompts and metacognitive prompts, by drawing on Bloom’s Taxonomy [62]. Cognitive prompts are goal-oriented, strategic inquiries that learners feed into a generative AI chatbot. Metacognitive prompts, on the other hand, are to foster learners’ learning judgement and metacognitive growth. For example, in the case of a writing class, a cognitive prompt could be, “Help me grasp the concept of a thesis statement”. An outcome-based prompt might be, “Revise the following sentence for clarity”. In the case of metacognitive prompts, a teacher could encourage the students to reflect on their essays by asking the AI chatbot, “Evaluate my essay and suggest improvements”. The AI chatbot may function as a writing consultant that provides feedback. Undeniably, students might take a quicker route by framing the process more “outcome-oriented”, such as asking the AI, “Refine and improve this essay”. This is where the educator’s role comes in to explain the ethics of conduct and its associated consequences. Self-regulated learners stand as ethical AI users who care about the learning journey, valuing more than just the end product. In summary, goals, outcomes, or objectives can be utilized as defined learning pathways (also known as prompts) when students interact with chatbots. Students defining goals while working with a chatbot can be seen as setting a parameter for their learning. This goal defining (or prompting) helps students clearly understand what they are expected to achieve during a learning session and facilitates their work self-assessment while working with a chatbot.

3.3. Feedback and Self-Assessment Mechanism

Self-assessment is a process in which individuals evaluate their learning, performance, and understanding of a particular subject or skill. Research has shown that self-assessment can positively impact learning outcomes, motivation, and metacognitive skills [63,64,65]. Specifically, self-assessment can help learners identify their strengths and weaknesses, re-set goals, and monitor their progress toward achieving those goals. Self-assessment, grounded in JOL, involves learners reflecting on their learning and making judgements about their level of understanding and progress [66]. Self-assessment is also a component of SRL, as it allows learners to monitor their progress and adjust their learning strategies or learning goals as needed [67]. Self-assessment can therefore be a feature of a chatbot regardless of whether learners employ it to self-assess their learning, or it can be automatically promoted by the chatbot system to guide students to self-assess.

However, so far, we have found that the current AI-powered chatbots, like ChatGPT, have limited capabilities in reverse prompting when used for educational purposes. Reverse prompting functions as guiding questions after students prompt the chatbot. As suggested in the last section, after learners identify their prompts and goals, chatbots can ask learners to reflect on their learning and provide “reverse prompts” for self-assessment. The concept of reverse prompts is similar to reciprocal questioning. Reciprocal questioning is a group-based process in which two students pose their own questions for each other to answer [68]. This method has been used mainly to facilitate the reading process for emergent readers [69,70,71]. For instance, a chatbot could ask a learner an explanatory question like “Now, I give you two thesis statements you requested. Can you provide more examples of the relationship between the two statements of X and Y?” or “Can you provide more details on the requested speech or action?” as well as reflective questions like “How do you generalize this principle to similar cases?” to rate their understanding of a particular concept on a scale from 1 to 5 or to identify areas where they need more practice. We mock an example of such a conversation below in Figure 2.

The chatbot could then provide feedback and resources to help the learner improve in areas with potential knowledge gaps and low confidence levels. In this way, chatbots can be an effective tool for encouraging student self-assessment and SRL. A great body of evidence shows that the integrative effect of self-assessment and just-in-time feedback goes beyond understanding and learning new concepts and skills [72]. Goal-oriented and criteria-based self-assessment (e.g., self-explanation and reflection prompts) allows the learner to identify the knowledge gaps and misconceptions that often lead to incorrect conceptions or cognitive conflicts. Just-in-time feedback (i.e., the information provided by an agent/tutor in response to the diagnosed gap) can then act as a knowledge repair mechanism if the provided information is perceived as clear, logical, coherent, and applicable by the learner [73].

Based on Table 1 and the previous section on prompting and reverse prompting, teachers can also focus on facilitating learning judgement while teaching students to work with an AI chatbot. However, we propose that reverse prompting from an AI chatbot is also important so that educational values and SRL can be achieved.

According to Zimmerman [8], a chatbot is the social assistance that students can obtain. If the chatbot can provide reverse prompts that guide thinking, reflection, and self-assessment, students can then execute strategies that fit their goals and knowledge level. When learners engage in self-assessment activities, they are engaging in the process of making judgments about their learning. Throughout self-assessment, learners develop an awareness of their strengths and weaknesses, which can help them modify or set new goals. If they are satisfied with their goals, they can use their goals to monitor their progress and adjust their strategies as needed. This process also aligns with Zimmerman’s SRL model of self-control. At this phase, students can decide whether to go with what the chatbot suggests or if they need to take what they have and implement the suggestions that the chatbot provides. For example, a chatbot could reversibly ask learners to describe their strategies to solve a particular problem or reflect on what they have learned from a particular activity. This type of reflection can help learners become more aware of their learning processes and develop more effective strategies for learning [74,75]. Thus, the reverse interaction from chatbot to students provides an opportunity for developing self-awareness because learners become more self-directed or self-regulated and independent in their learning while working with the chatbot, which can lead to improved academic performance and overall success. Furthermore, by incorporating self-assessment prompts into educational chatbots, learners can receive immediate feedback and support as they engage in the self-assessment process, which can help to develop their metacognitive skills further and promote deeper learning.

3.4. Facilitating Self-Regulation: Personalization and Adaptation

Personalization and adaptation are unique characteristics of learning technology. When students engage with an LMS, the LMS platform inherently captures and records their behaviors and interactions. This can encompass actions such as page views, time allocation per page, link traversal, and page-specific operations. Even the act of composing content within a discussion forum can offer comprehensive trace data, such as temporal markers indicating the writing and conclusion of a discussion forum post, syntactic structures employed, discernible genre attributes, and lexical choices. This collection of traceable data forms the foundation for the subsequent generation of comprehensive learning analytics for learners, being manifested as either textual reports or information visualizations, both encapsulating a synthesis of pertinent insights regarding the students’ learning trajectories [76]. These fine-grained analytical outputs can fulfill a key role in furnishing students with a holistic overview of how they learn and what they learn, fostering opportunities for reflection, evaluation, and informed refinement of their learning tactics. Therefore, by using data-driven insights and algorithms described above, chatbots can be tailored to the individual needs of learners, providing personalized feedback and guidance that supports their unique learning goals and preferences. However, we believe that the current AI-powered chatbot is inadequate in education; in particular, chatbots thus far lack capabilities for learning personalization and adaptation. A chatbot, like ChatGPT, often acts as a knowledge giver unless a learner knows how to feed the prompts. Our framework repositions the role of educational AI chatbots from knowledge providers to facilitators in the learning process. By encouraging students to initiate interactions through prompts, the chatbot assumes the role of a learning partner that progressively understands the students’ requirements. As outlined in the preceding section, the chatbot possesses the capability to tactfully prompt learners when necessary, offering guidance and directions instead of outright solutions based on the given prompts.

Learner adaptation can be effectively facilitated through the utilization of learning analytics, which serves as a valuable method for collecting learner data and enhancing overall learning outcomes [75]. Chatbots have become more practical and intelligent by improving natural language, data mining, and machine-learning techniques. The chatbot could use the trace data collected on LMS to provide students with the best course of action. Data that the chatbot can collect from the LMS can include analysis of students’ time spent on a page, students’ clicking behaviors, deadlines set by the instructors, or prompts (goals) initiated by the students. For example, a student has not viewed their module assignment pages on a learning management system for a long time, but they request the chatbot to generate a sample essay for their assignments. Instead of giving the direct output of a sample essay, the chatbot can direct the student to view the assignment pages more closely (i.e., “It looks like you haven’t spent enough time on this page, I suggest you review this page before attempting to ask me to give you an essay”), as shown in Figure 3. In this way, learning analytics can also help learners take ownership of their learning by providing real-time feedback on their progress and performance. By giving learners access to their learning analytics, educators can empower students to actively learn and make informed decisions about improving their performance [75,77]. An example is shown in Figure 4. Therefore, through personalized and adaptive chatbot interactions, learners can receive feedback and resources that are tailored to their specific needs and performance, helping to improve their metacognitive skills and ultimately enhancing their overall learning outcomes.

4. Limitations

Lo’s [78] comprehensive rapid review indicates three primary limitations inherent in generative AI tools: 1. biased information, 2. constrained access to current knowledge, and 3. propensity for disseminating false information [78]. Baidoo-Anu and Ansa [79] underscore that the efficacy of generative AI tools is intricately linked to the training data that were fed into the tool, wherein the composition of training data can inadvertently contain biases that subsequently manifest in the AI-generated content, potentially compromising the neutrality, objectivity, and reliability of information imparted to student users. Moreover, the precision and accuracy of the information generated by generative AI tools further emerge as a key concern. Scholarly investigations have discovered several instances where content produced by ChatGPT has demonstrated inaccuracy and spuriousness, particularly when tasked with generating citations for academic papers [79,80].

Amidst these acknowledged limitations, our position leans toward an emphasis on students’ educational use of these tools, transcending the preoccupation with the tools’ inherent characteristics of bias, inaccuracy, or falsity. Based on our proposal, we want to develop students’ capacity for self-regulation and discernment when evaluating received information. Furthermore, educators bear an important role in guiding students on harnessing the potential of generative AI tools to enhance the learning process, instead of the generative AI tools can provide information akin to a textbook. This justifies the reason why we integrate Zimmerman’s SRL model, illustrating how the judicious incorporation of generative AI tools can foster students’ self-regulation, synergizing with the guidance of educators and the efficacy of instructional technology design.

5. Concluding Remarks

This paper explores how educational chatbots, or so-called conversational agents, can support student self-regulatory processes and self-evaluation in the learning process. As shown in Figure 5 below, drawing on Zimmerman’s SRL framework, we postulate that chatbot designers should consider pedagogical principles, such as goal setting and planning, self-assessment, and personalization, to ensure that the chatbot effectively supports student learning and improves academic performance. We suggest that such a chatbot could provide personalized feedback to students on their understanding of course material and promote self-assessment by prompting them to reflect on their learning process. We also emphasize the importance of establishing the pedagogical functions of chatbots to fit the actual purposes of education and supplement teacher instruction. The paper provides examples of successful implementations of educational chatbots that can inform SRL process as well as self-assessment and reflection based on JOL principles. Overall, this paper highlights the potential benefits of educational chatbots for personalized and interactive learning experiences while emphasizing the importance of considering pedagogical principles in their design. Educational chatbots may supplement classroom instruction by providing personalized feedback and prompting reflection on student learning progress. However, chatbot designers must carefully consider how these tools fit into existing pedagogical practices to ensure their effectiveness in supporting student learning.

Through the application of our framework, future researchers are encouraged to delve into three important topics of inquiry that can empirically validate our conceptual model. The first dimension entails scrutiny of educational principles. For instance, how can AI chatbots be designed to support learners in setting and pursuing personalized learning goals, fostering a sense of ownership over the learning process? Addressing this question involves exploring how learners can form a sense of ownership over their interactions with the AI chatbots, while working towards the learning objectives.

The second dimension involves a closer examination of the actual Self-Regulated Learning (SRL) process. This necessitates an empirical exploration of the ways AI chatbots can effectively facilitate learners’ self-regulated reflections and the honing of self-regulation skills. For example, how effective is AI’s feedback to a student’s essay and how do students develop subsequent SRL strategies to address the AI’s feedback and evaluation? Additionally, inquiries might also revolve around educators’ instructional methods in leveraging AI chatbots to not only nurture learners’ skills in interacting with the technology but also foster their self-regulatory processes. Investigating the extent to which AI chatbots can provide learning analytics as feedback that harmonizes with individual learners’ self-regulation strategies is also of significance. Moreover, ethical considerations must be taken into account when integrating AI chatbots into educational settings, ensuring the preservation of learners’ autonomy and self-regulation.

The third dimension is related to user interface research. A research endeavor could revolve around identifying which conversational interface proves the most intuitive for learners as they engage with an AI chatbot. Additionally, an inquiry might probe the extent to which the AI chatbot should engage in dialogue within educational contexts. Furthermore, delineating the circumstances under which AI chatbots should abstain from delivering outcome-based outputs to learners constitutes a worthwhile avenue of investigation. Numerous additional inquiries can be derived from our conceptual model, yet the central message that we want to deliver remains clear: Our objective is to engage educators, instructional designers, and students in the learning process while navigating in this AI world. It is important to educate students on the potential of AI chatbots to enhance their self-regulation skills while also emphasizing the importance of avoiding actions that contravene the principles of academic integrity.

Conceptualization, D.H.C. and M.P.-C.L.; writing—original draft preparation, D.H.C.; writing—review and editing, D.H.C., M.P.-C.L., S.H. and Q.Q.W.; visualization, Q.Q.W.; funding acquisition, D.H.C. All authors have read and agreed to the published version of the manuscript.

Not applicable.

The authors declare no conflict of interest.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

View Image - Figure 1. Zimmerman’s multi-level SRL Framework (adopted from Panadero [7]).

Figure 1. Zimmerman’s multi-level SRL Framework (adopted from Panadero [7]).

View Image - Figure 2. A mocked example of reverse prompting from a chatbot.

Figure 2. A mocked example of reverse prompting from a chatbot.

View Image - Figure 3. An example of a chatbot in a learning management system that supports SRL by delivering personalized feedback.

Figure 3. An example of a chatbot in a learning management system that supports SRL by delivering personalized feedback.

View Image - Figure 4. An example of a chatbot that supports SRL by delivering learning analytics.

Figure 4. An example of a chatbot that supports SRL by delivering learning analytics.

View Image - Figure 5. Putting it all together: educational principles, SRL, and directionality.

Figure 5. Putting it all together: educational principles, SRL, and directionality.

Types of prompts based on Bloom’s Taxonomy [ ].

Prompt Types Process-Based Outcome-Based
Cognitive UnderstandRemember CreateApply
Metacognitive Evaluate

1. Pérez, J.Q.; Daradoumis, T.; Puig, J.M.M. Rediscovering the use of chatbots in education: A systematic literature review. Comput. Appl. Eng. Educ. ; 2020; 28 , pp. 1549-1565. [DOI: https://dx.doi.org/10.1002/cae.22326]

2. Smutny, P.; Schreiberova, P. Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Comput. Educ. ; 2020; 151 , 103862. [DOI: https://dx.doi.org/10.1016/j.compedu.2020.103862]

3. Kuhail, M.A.; Alturki, N.; Alramlawi, S.; Alhejori, K. Interacting with educational chatbots: A systematic review. Educ. Inf. Technol. ; 2023; 28 , pp. 973-1018. [DOI: https://dx.doi.org/10.1007/s10639-022-11177-3]

4. Okonkwo, C.W.; Ade-Ibijola, A. Chatbots applications in education: A systematic review. Comput. Educ. Artif. Intell. ; 2021; 2 , 100033. [DOI: https://dx.doi.org/10.1016/j.caeai.2021.100033]

5. Koriat, A. Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning. J. Exp. Psychol. Gen. ; 1997; 126 , pp. 349-370. [DOI: https://dx.doi.org/10.1037/0096-3445.126.4.349]

6. Son, L.K.; Metcalfe, J. Judgments of learning: Evidence for a two-stage process. Mem. Cogn. ; 2005; 33 , pp. 1116-1129. [DOI: https://dx.doi.org/10.3758/BF03193217] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/16496730]

7. Panadero, E. A Review of Self-regulated Learning: Six Models and Four Directions for Research. Front. Psychol. ; 2017; 8 , 422. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28503157][DOI: https://dx.doi.org/10.3389/fpsyg.2017.00422]

8. Zimmerman, B.J. Attaining Self-Regulation. Handbook of Self-Regulation ; Elsevier: Amsterdam, The Netherlands, 2000; pp. 13-39. [DOI: https://dx.doi.org/10.1016/B978-012109890-2/50031-7]

9. Baars, M.; Wijnia, L.; de Bruin, A.; Paas, F. The Relation Between Students’ Effort and Monitoring Judgments During Learning: A Meta-analysis. Educ. Psychol. Rev. ; 2020; 32 , pp. 979-1002. [DOI: https://dx.doi.org/10.1007/s10648-020-09569-3]

10. Leonesio, R.J.; Nelson, T.O. Do different metamemory judgments tap the same underlying aspects of memory?. J. Exp. Psychol. Learn. Mem. Cogn. ; 1990; 16 , pp. 464-470. [DOI: https://dx.doi.org/10.1037/0278-7393.16.3.464]

11. Double, K.S.; Birney, D.P.; Walker, S.A. A meta-analysis and systematic review of reactivity to judgements of learning. Memory ; 2018; 26 , pp. 741-750. [DOI: https://dx.doi.org/10.1080/09658211.2017.1404111]

12. Janes, J.L.; Rivers, M.L.; Dunlosky, J. The influence of making judgments of learning on memory performance: Positive, negative, or both?. Psychon. Bull. Rev. ; 2018; 25 , pp. 2356-2364. [DOI: https://dx.doi.org/10.3758/s13423-018-1463-4]

13. Hamzah, M.I.; Hamzah, H.; Zulkifli, H. Systematic Literature Review on the Elements of Metacognition-Based Higher Order Thinking Skills (HOTS) Teaching and Learning Modules. Sustainability ; 2022; 14 , 813. [DOI: https://dx.doi.org/10.3390/su14020813]

14. Veenman, M.V.J.; Van Hout-Wolters, B.H.A.M.; Afflerbach, P. Metacognition and learning: Conceptual and methodological considerations. Metacognition Learn. ; 2006; 1 , pp. 3-14. [DOI: https://dx.doi.org/10.1007/s11409-006-6893-0]

15. Nelson, T.; Narens, L. Why investigate metacognition. Metacognition: Knowing about Knowing ; MIT Press: Cambridge, MA, USA, 1994; [DOI: https://dx.doi.org/10.7551/mitpress/4561.003.0003]

16. Tuysuzoglu, B.B.; Greene, J.A. An investigation of the role of contingent metacognitive behavior in self-regulated learning. Metacognition Learn. ; 2015; 10 , pp. 77-98. [DOI: https://dx.doi.org/10.1007/s11409-014-9126-y]

17. Bandura, A. Social Cognitive Theory: An Agentic Perspective. Asian J. Soc. Psychol. ; 1999; 2 , pp. 21-41. [DOI: https://dx.doi.org/10.1111/1467-839X.00024]

18. Bem, D.J. Self-Perception Theory. Advances in Experimental Social Psychology ; Berkowitz, L. Academic Press: Cambridge, MA, USA, 1972; Volume 6 , pp. 1-62. [DOI: https://dx.doi.org/10.1016/s0065-2601(08)60024-6]

19. Abu Shawar, B.; Atwell, E. Different measurements metrics to evaluate a chatbot system. Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies ; Rochester, NY, USA, 26 April 2007; pp. 89-96. [DOI: https://dx.doi.org/10.3115/1556328.1556341]

20. Turing, A.M. Computing machinery and intelligence. Mind ; 1950; 59 , pp. 433-460. [DOI: https://dx.doi.org/10.1093/mind/LIX.236.433]

21. Weizenbaum, J. ELIZA—A computer program for the study of natural language communication between man and machine. Commun. ACM ; 1966; 9 , pp. 36-45. [DOI: https://dx.doi.org/10.1145/365153.365168]

22. Wallace, R.S. The anatomy of A.L.I.C.E. Parsing the Turing Test: Philosophical and Methodological Issues in the Quest for the Thinking Computer ; Epstein, R.; Roberts, G.; Beber, G. Springer: Dordrecht, The Netherlands, 2009; pp. 181-210. [DOI: https://dx.doi.org/10.1007/978-1-4020-6710-5_13]

23. Hwang, G.-J.; Chang, C.-Y. A review of opportunities and challenges of chatbots in education. Interact. Learn. Environ. ; 2021; pp. 1-14. [DOI: https://dx.doi.org/10.1080/10494820.2021.1952615]

24. Yamada, M.; Goda, Y.; Matsukawa, H.; Hata, K.; Yasunami, S. A Computer-Supported Collaborative Learning Design for Quality Interaction. IEEE MultiMedia ; 2016; 23 , pp. 48-59. [DOI: https://dx.doi.org/10.1109/MMUL.2015.95]

25. Muniasamy, A.; Alasiry, A. Deep Learning: The Impact on Future eLearning. Int. J. Emerg. Technol. Learn. (iJET) ; 2020; 15 , pp. 188-199. [DOI: https://dx.doi.org/10.3991/ijet.v15i01.11435]

26. Bendig, E.; Erb, B.; Schulze-Thuesing, L.; Baumeister, H. The Next Generation: Chatbots in Clinical Psychology and Psychotherapy to Foster Mental Health—A Scoping Review. Verhaltenstherapie ; 2022; 32 , pp. 64-76. [DOI: https://dx.doi.org/10.1159/000501812]

27. Kennedy, C.M.; Powell, J.; Payne, T.H.; Ainsworth, J.; Boyd, A.; Buchan, I. Active Assistance Technology for Health-Related Behavior Change: An Interdisciplinary Review. J. Med. Internet Res. ; 2012; 14 , e80. [DOI: https://dx.doi.org/10.2196/jmir.1893]

28. Poncette, A.-S.; Rojas, P.-D.; Hofferbert, J.; Sosa, A.V.; Balzer, F.; Braune, K. Hackathons as Stepping Stones in Health Care Innovation: Case Study with Systematic Recommendations. J. Med. Internet Res. ; 2020; 22 , e17004. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32207691][DOI: https://dx.doi.org/10.2196/17004]

29. Ferrell, O.C.; Ferrell, L. Technology Challenges and Opportunities Facing Marketing Education. Mark. Educ. Rev. ; 2020; 30 , pp. 3-14. [DOI: https://dx.doi.org/10.1080/10528008.2020.1718510]

30. Behera, R.K.; Bala, P.K.; Ray, A. Cognitive Chatbot for Personalised Contextual Customer Service: Behind the Scene and beyond the Hype. Inf. Syst. Front. ; 2021; pp. 1-21. [DOI: https://dx.doi.org/10.1007/s10796-021-10168-y]

31. Crolic, C.; Thomaz, F.; Hadi, R.; Stephen, A.T. Blame the Bot: Anthropomorphism and Anger in Customer–Chatbot Interactions. J. Mark. ; 2022; 86 , pp. 132-148. [DOI: https://dx.doi.org/10.1177/00222429211045687]

32. Clarizia, F.; Colace, F.; Lombardi, M.; Pascale, F.; Santaniello, D. Chatbot: An education support system for student. CSS 2018: Cyberspace Safety and Security ; Castiglione, A.; Pop, F.; Ficco, M.; Palmieri, F. Lecture Notes in Computer Science Book Series; Springer International Publishing: Cham, Switzerland, 2018; Volume 11161 , pp. 291-302. [DOI: https://dx.doi.org/10.1007/978-3-030-01689-0_23]

33. Firat, M. What ChatGPT means for universities: Perceptions of scholars and students. J. Appl. Learn. Teach. ; 2023; 6 , pp. 57-63. [DOI: https://dx.doi.org/10.37074/jalt.2023.6.1.22]

34. Kim, H.-S.; Kim, N.Y. Effects of AI chatbots on EFL students’ communication skills. Commun. Ski. ; 2021; 21 , pp. 712-734.

35. Hill, J.; Ford, W.R.; Farreras, I.G. Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Comput. Hum. Behav. ; 2015; 49 , pp. 245-250. [DOI: https://dx.doi.org/10.1016/j.chb.2015.02.026]

36. Wu, E.H.-K.; Lin, C.-H.; Ou, Y.-Y.; Liu, C.-Z.; Wang, W.-K.; Chao, C.-Y. Advantages and Constraints of a Hybrid Model K-12 E-Learning Assistant Chatbot. IEEE Access ; 2020; 8 , pp. 77788-77801. [DOI: https://dx.doi.org/10.1109/ACCESS.2020.2988252]

37. Brandtzaeg, P.B.; Følstad, A. Why people use chatbots. INSCI 2017: Internet Science ; Kompatsiaris, I.; Cave, J.; Satsiou, A.; Carle, G.; Passani, A.; Kontopoulos, E.; Diplaris, S.; McMillan, D. Lecture Notes in Computer Science Book Series; Springer International Publishing: Cham, Switzerland, 2017; Volume 10673 , pp. 377-392. [DOI: https://dx.doi.org/10.1007/978-3-319-70284-1_30]

38. Deng, X.; Yu, Z. A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education. Sustainability ; 2023; 15 , 2940. [DOI: https://dx.doi.org/10.3390/su15042940]

39. de Quincey, E.; Briggs, C.; Kyriacou, T.; Waller, R. Student Centred Design of a Learning Analytics System. Proceedings of the 9th International Conference on Learning Analytics & Knowledge ; Tempe, AZ, USA, 4 March 2019; pp. 353-362. [DOI: https://dx.doi.org/10.1145/3303772.3303793]

40. Hattie, J. The black box of tertiary assessment: An impending revolution. Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research ; Ako Aotearoa: Wellington, New Zealand, 2009; pp. 259-275.

41. Wisniewski, B.; Zierer, K.; Hattie, J. The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research. Front. Psychol. ; 2020; 10 , 3087. [DOI: https://dx.doi.org/10.3389/fpsyg.2019.03087]

42. Winne, P.H. Cognition and metacognition within self-regulated learning. Handbook of Self-Regulation of Learning and Performance ; 2nd ed. Routledge: London, UK, 2017.

43. Serban, I.V.; Sankar, C.; Germain, M.; Zhang, S.; Lin, Z.; Subramanian, S.; Kim, T.; Pieper, M.; Chandar, S.; Ke, N.R. et al. A deep reinforcement learning chatbot. arXiv ; 2017; arXiv: 1709.02349

44. Shneiderman, B.; Plaisant, C. Designing the User Interface: Strategies for Effective Human-Computer Interaction ; 4th ed. Pearson: Boston, MA, USA, Addison Wesley: Hoboken, NJ, USA, 2004.

45. Abbasi, S.; Kazi, H. Measuring effectiveness of learning chatbot systems on student’s learning outcome and memory retention. Asian J. Appl. Sci. Eng. ; 2014; 3 , pp. 57-66. [DOI: https://dx.doi.org/10.15590/ajase/2014/v3i7/53576]

46. Winkler, R.; Soellner, M. Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. Acad. Manag. Proc. ; 2018; 2018 , 15903. [DOI: https://dx.doi.org/10.5465/AMBPP.2018.15903abstract]

47. Dwivedi, Y.K.; Kshetri, N.; Hughes, L.; Slade, E.L.; Jeyaraj, A.; Kar, A.K.; Baabdullah, A.M.; Koohang, A.; Raghavan, V.; Ahuja, M. et al. Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Int. J. Inf. Manag. ; 2023; 71 , 102642. [DOI: https://dx.doi.org/10.1016/j.ijinfomgt.2023.102642]

48. Dai, W.; Lin, J.; Jin, F.; Li, T.; Tsai, Y.S.; Gasevic, D.; Chen, G. Can large language models provide feedback to students? A case study on ChatGPT. 2023; preprint [DOI: https://dx.doi.org/10.35542/osf.io/hcgzj]

49. Lin, M.P.-C.; Chang, D. Enhancing post-secondary writers’ writing skills with a Chatbot: A mixed-method classroom study. J. Educ. Technol. Soc. ; 2020; 23 , pp. 78-92.

50. Zhu, C.; Sun, M.; Luo, J.; Li, T.; Wang, M. How to harness the potential of ChatGPT in education?. Knowl. Manag. E-Learn. ; 2023; 15 , pp. 133-152. [DOI: https://dx.doi.org/10.34105/j.kmel.2023.15.008]

51. Prøitz, T.S. Learning outcomes: What are they? Who defines them? When and where are they defined?. Educ. Assess. Eval. Account. ; 2010; 22 , pp. 119-137. [DOI: https://dx.doi.org/10.1007/s11092-010-9097-8]

52. Burke, J. Outcomes, Learning and the Curriculum: Implications for Nvqs, Gnvqs and Other Qualifications ; Routledge: London, UK, 1995; [DOI: https://dx.doi.org/10.4324/9780203485835]

53. Locke, E.A. New Developments in Goal Setting and Task Performance ; 1st ed. Routledge: London, UK, 2013; [DOI: https://dx.doi.org/10.4324/9780203082744]

54. Leake, D.B.; Ram, A. Learning, goals, and learning goals: A perspective on goal-driven learning. Artif. Intell. Rev. ; 1995; 9 , pp. 387-422. [DOI: https://dx.doi.org/10.1007/BF00849065]

55. Greene, J.A.; Azevedo, R. A Theoretical Review of Winne and Hadwin’s Model of Self-Regulated Learning: New Perspectives and Directions. Rev. Educ. Res. ; 2007; 77 , pp. 334-372. [DOI: https://dx.doi.org/10.3102/003465430303953]

56. Pintrich, P.R. A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students. Educ. Psychol. Rev. ; 2004; 16 , pp. 385-407. [DOI: https://dx.doi.org/10.1007/s10648-004-0006-x]

57. Schunk, D.H.; Greene, J.A. Handbook of Self-Regulation of Learning and Performance ; 2nd ed. In Educational Psychology Handbook Series Routledge: New York, NY, USA, Taylor & Francis Group: Milton Park, UK, 2018.

58. Chen, C.-H.; Su, C.-Y. Using the BookRoll e-book system to promote self-regulated learning, self-efficacy and academic achievement for university students. J. Educ. Technol. Soc. ; 2019; 22 , pp. 33-46.

59. Michailidis, N.; Kapravelos, E.; Tsiatsos, T. Interaction analysis for supporting students’ self-regulation during blog-based CSCL activities. J. Educ. Technol. Soc. ; 2018; 21 , pp. 37-47.

60. Paans, C.; Molenaar, I.; Segers, E.; Verhoeven, L. Temporal variation in children’s self-regulated hypermedia learning. Comput. Hum. Behav. ; 2019; 96 , pp. 246-258. [DOI: https://dx.doi.org/10.1016/j.chb.2018.04.002]

61. Morisano, D.; Hirsh, J.B.; Peterson, J.B.; Pihl, R.O.; Shore, B.M. Setting, elaborating, and reflecting on personal goals improves academic performance. J. Appl. Psychol. ; 2010; 95 , pp. 255-264. [DOI: https://dx.doi.org/10.1037/a0018478]

62. Krathwohl, D.R. A Revision of Bloom’s Taxonomy: An Overview. Theory Pract. ; 2002; 41 , pp. 212-218. [DOI: https://dx.doi.org/10.1207/s15430421tip4104_2]

63. Bouffard, T.; Boisvert, J.; Vezeau, C.; Larouche, C. The impact of goal orientation on self-regulation and performance among college students. Br. J. Educ. Psychol. ; 1995; 65 , pp. 317-329. [DOI: https://dx.doi.org/10.1111/j.2044-8279.1995.tb01152.x]

64. Javaherbakhsh, M.R. The Impact of Self-Assessment on Iranian EFL Learners’ Writing Skill. Engl. Lang. Teach. ; 2010; 3 , pp. 213-218. [DOI: https://dx.doi.org/10.5539/elt.v3n2p213]

65. Zepeda, C.D.; Richey, J.E.; Ronevich, P.; Nokes-Malach, T.J. Direct instruction of metacognition benefits adolescent science learning, transfer, and motivation: An in vivo study. J. Educ. Psychol. ; 2015; 107 , pp. 954-970. [DOI: https://dx.doi.org/10.1037/edu0000022]

66. Ndoye, A. Peer/self assessment and student learning. Int. J. Teach. Learn. High. Educ. ; 2017; 29 , pp. 255-269.

67. Schunk, D.H. Goal and Self-Evaluative Influences During Children’s Cognitive Skill Learning. Am. Educ. Res. J. ; 1996; 33 , pp. 359-382. [DOI: https://dx.doi.org/10.3102/00028312033002359]

68. King, A. Enhancing Peer Interaction and Learning in the Classroom Through Reciprocal Questioning. Am. Educ. Res. J. ; 1990; 27 , pp. 664-687. [DOI: https://dx.doi.org/10.3102/00028312027004664]

69. Mason, L.H. Explicit Self-Regulated Strategy Development Versus Reciprocal Questioning: Effects on Expository Reading Comprehension Among Struggling Readers. J. Educ. Psychol. ; 2004; 96 , pp. 283-296. [DOI: https://dx.doi.org/10.1037/0022-0663.96.2.283]

70. Newman, R.S. Adaptive help seeking: A strategy of self-regulated learning. Self-Regulation of Learning and Performance: Issues and Educational Applications ; Lawrence Erlbaum Associates, Inc.: Hillsdale, NJ, USA, 1994; pp. 283-301.

71. Rosenshine, B.; Meister, C. Reciprocal Teaching: A Review of the Research. Rev. Educ. Res. ; 1994; 64 , pp. 479-530. [DOI: https://dx.doi.org/10.3102/00346543064004479]

72. Baleghizadeh, S.; Masoun, A. The Effect of Self-Assessment on EFL Learners’ Self-Efficacy. TESL Can. J. ; 2014; 31 , 42. [DOI: https://dx.doi.org/10.18806/tesl.v31i1.1166]

73. Moghadam, S.H. What Types of Feedback Enhance the Effectiveness of Self-Explanation in a Simulation-Based Learning Environment?. Available online: https://summit.sfu.ca/item/34750 (accessed on 14 July 2023).

74. Vanichvasin, P. Effects of Visual Communication on Memory Enhancement of Thai Undergraduate Students, Kasetsart University. High. Educ. Stud. ; 2020; 11 , pp. 34-41. [DOI: https://dx.doi.org/10.5539/hes.v11n1p34]

75. Schumacher, C.; Ifenthaler, D. Features students really expect from learning analytics. Comput. Hum. Behav. ; 2018; 78 , pp. 397-407. [DOI: https://dx.doi.org/10.1016/j.chb.2017.06.030]

76. Marzouk, Z.; Rakovic, M.; Liaqat, A.; Vytasek, J.; Samadi, D.; Stewart-Alonso, J.; Ram, I.; Woloshen, S.; Winne, P.H.; Nesbit, J.C. What if learning analytics were based on learning science?. Australas. J. Educ. Technol. ; 2016; 32 , pp. 1-18. [DOI: https://dx.doi.org/10.14742/ajet.3058]

77. Akhtar, S.; Warburton, S.; Xu, W. The use of an online learning and teaching system for monitoring computer aided design student participation and predicting student success. Int. J. Technol. Des. Educ. ; 2015; 27 , pp. 251-270. [DOI: https://dx.doi.org/10.1007/s10798-015-9346-8]

78. Lo, C.K. What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Educ. Sci. ; 2023; 13 , 410. [DOI: https://dx.doi.org/10.3390/educsci13040410]

79. Baidoo-Anu, D.; Ansah, L.O. Education in the era of generative Artificial Intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. SSRN Electron. J. ; 2023; pp. 1-22. [DOI: https://dx.doi.org/10.2139/ssrn.4337484]

80. Mogali, S.R. Initial impressions of ChatGPT for anatomy education. Anat. Sci. Educ. ; 2023; pp. 1-4. [DOI: https://dx.doi.org/10.1002/ase.2261] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/36749034]

You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer

Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

The invention of ChatGPT and generative AI technologies presents educators with significant challenges, as concerns arise regarding students potentially exploiting these tools unethically, misrepresenting their work, or gaining academic merits without active participation in the learning process. To effectively navigate this shift, it is crucial to embrace AI as a contemporary educational trend and establish pedagogical principles for properly utilizing emerging technologies like ChatGPT to promote self-regulation. Rather than suppressing AI-driven tools, educators should foster collaborations among stakeholders, including educators, instructional designers, AI researchers, and developers. This paper proposes three key pedagogical principles for integrating AI chatbots in classrooms, informed by Zimmerman’s Self-Regulated Learning (SRL) framework and Judgment of Learning (JOL). We argue that the current conceptualization of AI chatbots in education is inadequate, so we advocate for the incorporation of goal setting (prompting), self-assessment and feedback, and personalization as three essential educational principles. First, we propose that teaching prompting is important for developing students’ SRL. Second, configuring reverse prompting in the AI chatbot’s capability will help to guide students’ SRL and monitoring for understanding. Third, developing a data-driven mechanism that enables an AI chatbot to provide learning analytics helps learners to reflect on learning and develop SRL strategies. By bringing in Zimmerman’s SRL framework with JOL, we aim to provide educators with guidelines for implementing AI in teaching and learning contexts, with a focus on promoting students’ self-regulation in higher education through AI-assisted pedagogy and instructional design.

VIAFID ORCID Logo

Suggested sources

  • About ProQuest
  • Terms of Use
  • Privacy Policy
  • Cookie Policy

Designing an Educational Chatbot: A Case Study of CikguAIBot

  • Publisher site Copy service {{ this.onShowCLink("http://jdream3.com/copy/?sid=JGLOBAL&noSystem=1&documentNoArray=21A2304938&COPY=1") }}
  • Access JDreamⅢ for advanced search and analysis. {{ this.onShowJLink("http://jdream3.com/lp/jglobal/index.html?docNo=21A2304938&from=J-GLOBAL&jstjournalNo=W2441A") }}

About Nasharuddin Nurul Amelina

  • Search "202050000615927360"
  • Search "Nasharuddin Nurul Amelina"

About Universiti Putra Malaysia, UPM Serdang,Faculty of Computer Science and Information Technology,Selangor,Malaysia

  • Search "201551000096793320"
  • Search "Universiti Putra Malaysia, UPM Serdang,Faculty of Computer Science and Information Technology,Selangor,Malaysia"

About Sharef Nurfadhlina Mohd

  • Search "202150000726147965"
  • Search "Sharef Nurfadhlina Mohd"

About Mansor Evi Indriasari

  • Search "202150000709379151"
  • Search "Mansor Evi Indriasari"

About Universiti Putra Malaysia, UPM Serdang,Faculty of Educational Studies,Selangor,Malaysia

  • Search "Universiti Putra Malaysia, UPM Serdang,Faculty of Educational Studies,Selangor,Malaysia"

About Samian Normalia

  • Search "202150000728032936"
  • Search "Samian Normalia"

About Murad Masrah Azrifah Azmi

  • Search "202150000761463683"
  • Search "Murad Masrah Azrifah Azmi"

About Omar Mohd Khaizer

  • Search "202150000757475960"
  • Search "Omar Mohd Khaizer"

About Arshad Noreen Izza

  • Search "202150000721597008"
  • Search "Arshad Noreen Izza"

About Universiti Teknologi PETRONAS,Institute of Autonomous Systems,Positive Computing Research Center,Bandar Seri Iskandar,Perak,Malaysia

  • Search "201951000138506504"
  • Search "Universiti Teknologi PETRONAS,Institute of Autonomous Systems,Positive Computing Research Center,Bandar Seri Iskandar,Perak,Malaysia"

About Shahbodin Faaizah

  • Search "202150000726169608"
  • Search "Shahbodin Faaizah"

About Universiti Teknikal Malaysia Melaka,Centre for Academic Excellence and Scholarship,Hang Tuah Jaya,Melaka,Malaysia

  • Search "201551000065848984"
  • Search "Universiti Teknikal Malaysia Melaka,Centre for Academic Excellence and Scholarship,Hang Tuah Jaya,Melaka,Malaysia"

About Marhaban Mohammad Hamiruce

  • Search "202150000756728865"
  • Search "Marhaban Mohammad Hamiruce"

About Vice Chancellor Office, Universiti Sains Islam Malaysia,Bandar Baru Nilai,Negeri Sembilan,Malaysia

  • Search "202151000170158680"
  • Search "Vice Chancellor Office, Universiti Sains Islam Malaysia,Bandar Baru Nilai,Negeri Sembilan,Malaysia"

About IEEE Conference Proceedings

  • Search "W2441A"
  • Detailed information

Please login to MyJ-GLOBAL to see full information. You also need to select "Display abstract, etc. of medical articles" in your MyJ-GLOBAL account page in order to see abstracts, etc. of medical articles.

About education

  • Search "education"

About interaction

  • Search "interaction"

About language

  • Search "language"

About Malaysia

  • Search "Malaysia"

About artificial intelligence

  • Search "artificial intelligence"

About application program

  • Search "application program"

About case study

  • Search "case study"

About agent

  • Search "agent"

About 教育システム

  • Search "教育システム"
  • Search "学習者"

About チャットシステム

  • Search "チャットシステム"
  • Search "ボット"

About Graphic and image processing in general

  • Search "JE04010I"
  • Search "JE04"
  • Search "教育"

About チャットボット

  • Search "チャットボット"
  • Search "設計"
  • Search "研究"

designing an educational chatbot a case study of cikguaibot

Copyright (c) 2009 Japan Science and Technology Agency. All Rights Reserved

TRACE: A Conceptual Model to Guide the Design of Educational Chatbots

  • Conference paper
  • First Online: 01 February 2024
  • Cite this conference paper

designing an educational chatbot a case study of cikguaibot

  • Juan Carlos Farah 13 , 14 ,
  • Basile Spaenlehauer 13 ,
  • Sandy Ingram 14 ,
  • Fanny Kim-Lan Lasne 13 ,
  • María Jesús Rodríguez-Triana 15 ,
  • Adrian Holzer 16 &
  • Denis Gillet 13  

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 899))

Included in the following conference series:

  • International Conference on Interactive Collaborative Learning

126 Accesses

Driven by the rising popularity of chatbots such as ChatGPT, there is a budding line of research proposing guidelines for chatbot design, both in general and specifically for digital education. Nevertheless, few researchers have focused on providing conceptual tools to frame the chatbot design process itself. In this paper, we present a model to guide the design of educational chatbots. Our model aims to structure participatory design sessions in which different stakeholders (educators, developers, and learners) collaborate in the ideation of educational chatbots. To validate our model, we conducted an illustrative study in which 25 software design students took part in a simulated participatory design session. Students were divided into eight groups, assigned the role of one of the different stakeholders, and instructed to use our model. The results of our qualitative analysis suggest that our model helped structure the design process and align the contributions of the various stakeholders.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
  • Available as EPUB and PDF
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Bahja, M., Hammad, R., Butt, G.: A user-centric framework for educational Chatbots design and development. In: Stephanidis, C., Kurosu, M., Degen, H., Reinerman-Jones, L. (eds.) HCI International 2020—Late Breaking Papers: Multimodality and Intelligence, vol. 12424, pp. 32–43. Springer, Cham, Switzerland (2020). https://doi.org/10.1007/978-3-030-60117-1_3

Dillman, K.R., Mok, T.T.H., Tang, A., Oehlberg, L., Mitchell, A.: A visual interaction cue framework from video game environments for augmented reality. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–12. ACM, Montreal, QC, Canada (2018). https://doi.org/10.1145/3173574.3173714

Durall Gazulla, E., Martins, L., Fernández-Ferrer, M.: Designing learning technology collaboratively: analysis of a chatbot co-design. Educ. Inf. Technol. 28 (1), 109–134 (2023). https://doi.org/10.1007/s10639-022-11162-w

Article   Google Scholar  

Farah, J.C., Ingram, S., Gillet, D.: Supporting developers in creating web apps for education via an app development framework. In: HEAd’22 Conference Proceedings, pp. 883–890. Editorial Universitat Politècnica de València, Valencia, Spain (2022). https://doi.org/10.4995/HEAd22.2022.14587

Farah, J.C., Spaenlehauer, B., Ingram, S., Gillet, D.: A blueprint for integrating task-oriented conversational agents in education. In: Proceedings of the 4th Conference on Conversational User Interfaces, CUI’22. ACM, Glasgow, United Kingdom (2022). https://doi.org/10.1145/3543829.3544525

Feine, J., Morana, S., Maedche, A.: Designing a Chatbot social cue configuration system. In: Proceedings of the 40th International Conference on Information Systems (ICIS) (2019)

Google Scholar  

Feine, J., Morana, S., Maedche, A.: Designing interactive Chatbot development systems. In: Proceedings of the 41st International Conference on Information Systems (ICIS) (2020)

Felten, E., Raj, M., Seamans, R.: How will language modelers like ChatGPT affect occupations and industries? (2023)

Ferman Guerra, M.A.: Towards best practices for Chatbots. Master’s thesis, University of Victoria, Victoria, BC, Canada (2018)

Griol, D., Callejas, Z.: An architecture to develop multimodal educative applications with Chatbots. Int. J. Adv. Robot. Syst. 10 (3) (2013). https://doi.org/10.5772/55791

Hu, X., Moore, A., Coleman Eubanks, J., Aiyaz, A., McMahan, R.P.: Evaluating interaction cue purpose and timing for learning and retaining virtual reality training. In: Proceedings of the 2020 ACM Symposium on Spatial User Interaction, SUI ’20. ACM, Virtual Event, Canada (2020). https://doi.org/10.1145/3385959.3418448

Jung, H., Lee, J., Park, C.: Deriving design principles for educational chatbots from empirical studies on human-chatbot interaction. J. Digi. Contents Soc. 21 (3), 487–493 (2020). https://doi.org/10.9728/dcs.2020.21.3.487

Kuhail, M.A., Alturki, N., Alramlawi, S., Alhejori, K.: Interacting with educational chatbots: a systematic review. Educ. Inf. Technol. 28 (1), 973–1018 (2023). https://doi.org/10.1007/s10639-022-11177-3

Nass, C., Steuer, J., Tauber, E.R.: Computers are social actors. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’94, pp. 72–78. ACM, Boston, MA, USA (1994). https://doi.org/10.1145/191666.191703

OpenAI: Introducing ChatGPT (2022). https://openai.com/blog/chatgpt

Quiroga Pérez, J., Daradoumis, T., Marquès Puig, J.M.: Rediscovering the use of chatbots in education: a systematic literature review. Comput. Appl. Eng. Educ. 28 (6), 1549–1565 (2020). https://doi.org/10.1002/cae.22326

Richter, S.: Learning tasks. In: Seel, N.M. (ed.) Encyclopedia of the Sciences of Learning, pp. 1975–1979. Springer, Boston, MA, USA (2012). https://doi.org/10.1007/978-1-4419-1428-6_342

Rodríguez-Triana, M.J., Martínez-Monés, A., Villagrá-Sobrino, S.: Learning analytics in small-scale teacher-led innovations: ethical and data privacy issues. J. Learn. Anal. 3 (1) (2016). https://doi.org/10.18608/jla.2016.31.4

Vector Stall: Assistant free icons. https://www.flaticon.com/free-icon/assistant/_4818971

Wiley, D.A.: Connecting learning objects to instructional design theory: a definition, a metaphor, and a taxonomy. In: The Instructional Use of Learning Objects, pp. 3–23. AIT/AECT (2002)

Download references

Author information

Authors and affiliations.

École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

Juan Carlos Farah, Basile Spaenlehauer, Fanny Kim-Lan Lasne & Denis Gillet

University of Applied Sciences (HES-SO), Fribourg, Switzerland

Juan Carlos Farah & Sandy Ingram

Tallinn University, Tallinn, Estonia

María Jesús Rodríguez-Triana

University of Neuchâtel, Neuchâtel, Switzerland

Adrian Holzer

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Juan Carlos Farah .

Editor information

Editors and affiliations.

CTI Global, Frankfurt, Germany

Michael E. Auer

UTN—FRBA, Mozart, Argentina

Uriel R. Cukierman

DISA, Technical University of Valencia, Valencia, Spain

Eduardo Vendrell Vidal

UPM, ETSII, Madrid, Spain

Edmundo Tovar Caro

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper.

Farah, J.C. et al. (2024). TRACE: A Conceptual Model to Guide the Design of Educational Chatbots. In: Auer, M.E., Cukierman, U.R., Vendrell Vidal, E., Tovar Caro, E. (eds) Towards a Hybrid, Flexible and Socially Engaged Higher Education. ICL 2023. Lecture Notes in Networks and Systems, vol 899. Springer, Cham. https://doi.org/10.1007/978-3-031-51979-6_46

Download citation

DOI : https://doi.org/10.1007/978-3-031-51979-6_46

Published : 01 February 2024

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-51978-9

Online ISBN : 978-3-031-51979-6

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publish with us

Policies and ethics

  • Find a journal
  • Track your research

IEEE Account

  • Change Username/Password
  • Update Address

Purchase Details

  • Payment Options
  • Order History
  • View Purchased Documents

Profile Information

  • Communications Preferences
  • Profession and Education
  • Technical Interests
  • US & Canada: +1 800 678 4333
  • Worldwide: +1 732 981 0060
  • Contact & Support
  • About IEEE Xplore
  • Accessibility
  • Terms of Use
  • Nondiscrimination Policy
  • Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

  • Research article
  • Open access
  • Published: 19 December 2023

Integrating chatbots in education: insights from the Chatbot-Human Interaction Satisfaction Model (CHISM)

  • Jose Belda-Medina   ORCID: orcid.org/0000-0003-0115-0417 1 &
  • Vendula Kokošková 2  

International Journal of Educational Technology in Higher Education volume  20 , Article number:  62 ( 2023 ) Cite this article

4270 Accesses

4 Citations

3 Altmetric

Metrics details

Recent advances in Artificial Intelligence (AI) have paved the way for the integration of text-based and voice-enabled chatbots as adaptive virtual tutors in education. Despite the increasing use of AI-powered chatbots in language learning, there is a lack of studies exploring the attitudes and perceptions of teachers and students towards these intelligent tutors. This study aims to compare several linguistic and technological aspects of four App-Integrated Chatbots (AICs) and to examine the perceptions among English as a Foreign Language (EFL) teacher candidates. In this mixed-methods research based on convenience sampling, 237 college students from Spain (n = 155) and the Czech Republic (n = 82) interacted with four AICs over a month, and evaluated them following a rubric based on the Chatbot-Human Interaction Satisfaction Model. This scale was specifically designed to assess different linguistic and technological features of AICs such as response interval, semantic coherence, sentence length, and user interface. Quantitative and qualitative data were gathered through a pre-post-survey, based on the CHISM model and student assessment reports. Quantitative data were analyzed using SPSS statistics software, while qualitative data were examined using QDA Miner software, focusing on identifying recurring themes through frequency analysis. The findings indicated a moderate level of satisfaction with AICs, suggesting that enhancements in areas such as better adapting to learner needs, integrating interactive multimedia, and improving speech technologies are necessary for a more human-like user interaction.

Introduction

Chatbot technology has evolved rapidly over the last 60 years, partly thanks to modern advances in Natural Language Processing (NLP) and Machine Learning (ML) and the availability of Large Language Models (LLMs). Today chatbots can understand natural language, respond to user input, and provide feedback in the form of text or audio (text-based and voice-enabled). They can offer learners the possibility to engage in simulated conversational interactions in a non-judgmental environment (El Shazly, 2021 ; Skjuve et al., 2021 ). For these reasons, chatbots are being increasingly used as virtual tutors to facilitate the development of language skills and communicative competence in the target language (Huang et al., 2022 ; Hwang & Chang, 2021 ; Zhang et al., 2023 ).

Additionally, chatbots can be employed to furnish language learners with supplementary resources and provide immediate assistance such as access to online dictionaries, digital materials, and social media in real-time (Dokukina & Gumanova, 2020 ; Haristiani & Rifa’i, 2020 ). Modern chatbots can include speech technologies (Recognition and Synthesis or R&S) and can be customized to cater to the specific needs of individual learners, thus allowing for the provision of personalized feedback and adaptive support in the learning process (Jeon et al., 2023 ; Kim et al., 2019 ).

Thanks to these advances, the incorporation of chatbots into language learning applications has been on the rise in recent years (Fryer et al., 2020 ; Godwin-Jones, 2022 ; Kohnke, 2023 ). Their main goal is to engage learners in simulated conversations in a digital environment, providing interactive exercises to practice pronunciation, vocabulary, and grammar, detecting and correcting errors in real-time, and adapting the instruction to the individual learner’s needs. The wide accessibility of chatbots as virtual language tutors, regardless of temporal and spatial constraints, represents a substantial advantage over human instructors.

However, previous studies (Huang et al., 2022 ; Kim et al., 2019 ; Vera, 2023 ) have identified several limitations associated with the adoption of chatbots in language instruction such as the presence of redundancies (off-topics and prearranged answers), limited ability to understand more complex questions or sentence structures (sentence length and complexity), limited understanding of the contextual meaning and non-standard language (lexical richness), and the inability to engage in multiuser conversational interactions. These limitations may result in learners' lack of engagement and satisfaction with the chatbot (Jeon, 2021 ). Nevertheless, these limitations often stem from the chatbot design and are not inherent flaws. As AI technology is rapidly evolving, some of these issues are being progressively addressed, enhancing chatbots' capabilities and potentially reducing learners' dissatisfaction over time.

Research on app-integrated chatbots (AICs henceforth) in language learning is relatively scarce compared to the extensive literature on mobile app usage in language learning (MALL), primarily due to the emerging nature of this AI technology. Consequently, the present study aims to investigate several linguistic and technological features of four AICs ( Mondly, Andy, John Bot, and Buddy.ai) and to examine teacher candidates’ perceptions toward them through an ad-hoc model named the Chatbot-Human Interaction Satisfaction Model (CHISM). The novelty of this research is twofold. Firstly, it aims to analyse EFL teacher candidates' perceptions and interest in AICs as current students and future educators. Secondly, it proposes an adapted model of CHISM (Belda-Medina et al., 2022 ) to examine learner satisfaction with AICs, considering both technological and linguistic dimensions, which can be applied to future research. Our study specifically addresses the limitations previously mentioned in chatbot integration for language learning by focusing on the user's experience from three distinct perspectives: language, design, and user interaction. Through the application of the CHISM model, we comprehensively evaluate these aspects, setting the foundation for future research in language education.

The study has three main objectives. Firstly, it aims to investigate the current knowledge and opinions of language teacher candidates regarding App-Integrated Chatbots (AICs). Secondly, it seeks to measure their level of satisfaction with four specific AICs after a 1-month intervention. Lastly, it aims to evaluate their perspectives on the potential advantages and drawbacks of AICs in language learning as future educators.

Literature review

Chatbot definition.

A chatbot, short for chatterbot, is a computer program that uses artificial intelligence (AI) to conduct a conversation via auditory or textual methods and interacts with humans in their natural languages. These interactions usually occur through websites, messaging applications, or mobile apps, where the bot is capable of simulating and maintaining human-like conversations and perform different tasks (Adamopoulou & Moussiades, 2020 ).

In this research, the term chatbot (AIC) is used to refer to virtual tutors integrated into mobile applications specifically designed for language learning to provide students with a personalized and interactive experience. These AICs may cover different aspects of language learning, such as grammar, vocabulary, pronunciation, and listening comprehension, and use various techniques to adapt to the user's level of proficiency and tailor their responses accordingly.

App-Integrated Chatbots (AICs) in language learning

The proliferation of smartphones in the late 2000s led to the integration of educational chatbots into mobile applications. However, the initial models were basic, relying on a scripted question–answer format and not intended for meaningful practice beyond their specific subject area (Godwin-Jones, 2022 ). Since then, AI technology has significantly advanced and chatbots are now able to provide more comprehensive language learning support, such as conversational exchange, interactive activities, and multimedia content (Jung, 2019 ; Li et al., 2022 ).

Existing literature on AIC integration focuses on three main areas. The first one delves into the effects of AICs on language competence and skills. Research in this area probes the efficacy of chatbots in fostering significant improvements in different linguistic aspects, including but not limited to grammar (Haristiani et al., 2019 ; Kharis et al., 2022 ; Kim, 2019 ), vocabulary (Ajisoko, 2020 ; Kim, 2020 ), writing (Pin-Chuan Lin & Chang, 2020 ), and conversation (Hakim & Rima, 2022 ; Pham et al., 2018 ). These studies showed how AICs can manage personal queries, correct language mistakes, and offer linguistic support in real-time. However, some authors noted certain limitations, such as their reliance on large data for learning and prediction, and their potential inability to understand different accents, language nuances, and context, which could lead to conversational errors (Huang et al., 2022 ; Panesar, 2020 ).

.The second major theme is the impact of AICs on learner engagement and motivation (Dokukina & Gumanova, 2020 ; Li et al., 2022 ). This line of research investigates how the interactive nature of some AICs can reduce students’ anxiety and cognitive load (Hsu et al., 2021 ) and promote an engaging learning environment (Bao, 2019 ). Furthermore, some authors have examined the ability of chatbots to promote self-directed learning, given their wide availability and capacity for personalized responses (Annamalai et al., 2023 ). Nonetheless, certain researchers, including Ayedoun et al. ( 2015 ) and Fryer et al. ( 2019 ), have indicated that the initial enthusiasm and engagement students show towards chatbots may be short-lived, attributing this to the novelty effect of this technology.

The third area explores how AICs' design can positively affect language learning outcomes. Modern AICs usually include an interface with multimedia content, real-time feedback, and social media integration (Haristiani & Rifa’I, 2020 ). They also employ advanced speech technologies to ensure accessible and humanlike dialogues (Petrović & Jovanović, 2021 ). Additionally, AICs today can also incorporate emerging technologies like AR and VR, and gamification elements, to enhance learner motivation and engagement (Kim et al., 2019 ). However, some studies have also noted certain limitations, such as repetitive exercises and imperfect speech technologies, suggesting they should be used only as supplementary tools (Jung, 2019 ) or assistants (Kukulska-Hulme & Lee, 2020 ) rather than intelligent virtual tutors.

Teachers and learners’ views on the use of AICs for language learning

The landscape of mobile-application language learning (MALL) has been significantly reshaped in recent years with the incorporation of AICs (Pham et al., 2018 ). This innovative approach to mobile learning has been positively received by both students and teachers. For example, Chen et al. ( 2020 ) highlighted the effectiveness of AICs for Chinese vocabulary learning by comparing chatbot-based tutoring with traditional classroom settings. The study reported positive user feedback on the chatbot's ease of use, usefulness, and enjoyment, as measured by the Technology Acceptance Model (TAM). Similarly, Yang ( 2022 ) underscored the favourable views of AICs in English language education, with teachers valuing the chatbot’s capacity to manage routine tasks, thereby allowing them to concentrate on more substantial classroom duties. In this study, students appreciated the supplemental use of chatbots for their ability to provide immediate feedback on unfamiliar words or concepts, thereby enriching their English textbook learning.

However, the use of AICs as virtual tutors also presents certain challenges. Some studies have emphasized that interactions with AICs can seem detached and lack the human element (Rapp et al., 2021 ). Additionally, while AICs can handle a wide range of queries, they may struggle with complex language nuances, which could potentially lead to misunderstandings or incorrect language usage. It has also been observed that some students' interest dwindled after the initial period of engagement due to repetitive conversation patterns and redundancies, making the interaction less natural compared to student–teacher exchanges (Fryer et al., 2019 ).

In our study, the term 'perceptions' is defined, following Chuah and Kabilan’s approach (2021), as users' attitudes and opinions towards their interactions with chatbots in education. This encompasses aspects such as perceived usefulness, acceptance, and potential interest. Research in this area underscores the importance of understanding users' viewpoints on chatbots, including their acceptance of these tools in educational settings and their preferences for chatbot-human communication. Similarly, 'satisfaction' is described as the degree to which users feel that their needs and expectations are met by the chatbot experience, encompassing both linguistic and design aspects. Studies like those by Chocarro et al. ( 2023 ) have delved into students' enjoyment and engagement with chatbots, highlighting the importance of bot proactiveness and individual user characteristics in shaping students' satisfaction with chatbots in educational settings.

Despite these insights, there remains a significant gap in the literature regarding a comprehensive understanding of teachers’ and students’ perceptions of AICs, particularly in how these perceptions influence their acceptance and effectiveness in language education. This gap is more pronounced in understanding how the design and linguistic features of AICs impact user satisfaction and engagement. While studies like those of Chen et al. ( 2020 ) and Chocarro et al. ( 2023 ) have begun exploring these areas, there is a need for a more targeted framework to evaluate satisfaction with AICs in the context of language learning. To address this need, our study investigates EFL teacher candidates’ levels of satisfaction and perceptions of four AICs. We propose the Chatbot-Human Interaction Satisfaction Model (CHISM), an adaptation from a previous model used with intelligent conversational agents (Belda-Medina et al., 2022 ), to specifically measure and analyze these perceptions and their impact on language learning among language teacher candidates.

Questions and objectives

The study is structured around three research questions:

What prior knowledge do teacher candidates have about using App-Integrated Chatbots (AICs) in language learning?

How satisfied are participants with the four AICs selected after a 1-month intervention?

What potential benefits and limitations of AICs do teacher candidates perceive in language learning?

These questions align with three objectives. The first objective explores the existing knowledge and understanding of language teacher candidates about AICs in language learning. This includes their familiarity with chatbot technologies, perceptions of their utility, and any prior experiences they might have with these tools. The second objective assesses participants' satisfaction levels with four selected AICs after a 1-month intervention, utilizing the Chatbot-Human Language Interaction Satisfaction Model (CHISM) to evaluate linguistic accuracy and user interface design. The third objective evaluates the participants' perceptions as future educators of the potential benefits and limitations of AICs in language learning. This objective considers their views on how AICs might be integrated into future language education settings, highlighting their potential impact on teaching and learning methodologies. The corresponding research objectives are as follows:

O1. Explore the prior knowledge of language teacher candidates regarding chatbots.

O2. Measure their level of satisfaction after the intervention with four AICs through the Chatbot-Human Language Interaction Satisfaction Model (CHISM).

O3. Evaluate the participants' perceptions of AICs in language learning as future educators.

Participants and context

The research, conducted over two academic years (2020–2022) with a mixed-methods approach and convenience sampling, initially involved 163 students from the University of X (Spain) and 86 from the University of X (Czech Republic). However, the final participant count was 155 Spanish students and 82 Czech students, as some declined to participate or did not submit the required tasks. Participation was voluntary, and students who actively engaged with the chatbots and completed all tasks, including submitting transcripts and multiple-date screenshots, were rewarded with extra credits in their monthly quizzes. This approach ensured higher participation and meaningful interaction with the chatbots, contributing to the study's insights into the effectiveness of AICs in language education.

Participants were third-year-college students enrolled in two subjects on Applied Linguistics taught over the course of 4 months, with two-hour sessions being held twice a week. Both Applied Linguistics courses are integral components of the Teacher Education degree programs at the respective universities in Spain and the Czech Republic. These participants were being trained to become English language teachers, and the learning module on chatbot integration into language learning was strategically incorporated into the syllabus of both subjects, taught by the researchers. The choice of Spain and the Czech Republic was primarily based on convenience sampling. The two researchers involved in this study are also lecturers at universities in these respective countries, which facilitated access to a suitable participant pool. Additionally, the decision to include these two different educational settings aimed to test the applicability and effectiveness of AICs across varied contexts. The study found similar results in both settings, strengthening the argument for the broader relevance and potential of AICs in diverse educational environments.

The language proficiency of the students aligned with the upper intermediate (B2) and advanced (C1) levels as defined by the Common European Framework of Reference for Languages (CEFR), while some participants were at the native speaker (C2) level. In our study, the primary focus was on evaluating language teacher candidates' perceptions of AICs in language learning, rather than assessing language learning outcomes. Considering that the majority of participants possessed an upper intermediate (B2-C1) or advanced (C2) proficiency level, the distinction between native and non-native speakers was not deemed a crucial factor for this research. Subsequently, a statistical analysis was conducted to evaluate the impact of language nativeness (Spanish and Czech versus non-Spanish and non-Czech speakers), revealing no significant differences in the study’s outcomes. Furthermore, the evaluations of the AICs by both Spanish and Czech cohorts displayed similar results. This analysis led us to conclude that language nativeness and the specific educational settings of the participants were not key factors influencing the results of our study. Regarding gender, 81% of the participants were females, while 19% were male students. All participants were under 30 years of age.

The research was carried out following the regulations set by each institution for interventions with human subjects, as approved by their respective Ethical Committees. Participants provided written consent for the publication of their interactions with chatbots for academic purposes. All data obtained were anonymised and analysed confidentially.

Instruments and procedure

Following a sequential QUAN-QUAL approach (Nastasi et al., 2007 ), data were gathered through a pre-post survey, class discussion and assessment reports. The pre-survey (15 items) was divided into two sections: a socio-demographic Section (3 items), and a technological affinity Section (12 items) that focused on participants' usage, prior knowledge, and perceptions toward chatbots. The post-survey (15 items) was designed to gather data related to the use of the four AICs ( Mondly, Andy, John Bot, Buddy.ai ) employed in the intervention. The study incorporated th.

The Chatbot-Human Interaction Satisfaction Model (CHISM) is a tool previously designed and used to measure participants' satisfaction with intelligent conversational agents in language learning (Belda-Medina et al., 2022 ). This model was specifically adapted for this study to be implemented with AICs. The pre-post surveys were completed in the classroom in an electronic format during class time to ensure a focused environment for the participants. Quantitative data obtained were analysed using the IBM ® SPSS ® Statistics software 27. The main objective was to determine the average responses by calculating the means, evaluate the variability in the data by measuring the standard deviation, and assess the distribution's flatness through kurtosis.

Qualitative data were collected through class discussions and assessment reports of the AICS following a template provided through the Moodle platform. During the 1-month intervention period in each educational setting, participants independently completed the assessment reports. They were instructed to provide personal feedback on their interaction with each AIC, using the template to note both positive and negative aspects. Additionally, they were asked to attach 12 screenshots illustrating their interaction, three with each AIC, to support their assessment. QDA Miner Software was used for textual analysis of students' written evaluations on each AIC, adhering to a provided template. Student comments were systematically categorized into potential benefits and limitations following the template structure and then coded using a tree-structured code system, focusing on recurrent themes through frequency analysis. The research comprised four stages as shown in Fig.  1 .

figure 1

Research procedure

For the interaction, detailed instructions were provided via Moodle, with the aim not to measure the participants’ English learning progress, but to enable critical analysis of each AIC as future educators. The teacher candidates were guided on how to engage with the chatbots, including selecting different language levels, using varied sentence types, introducing typical errors, exploring voice options, and investigating the use of AR and other technologies if available. This assessment was aligned with the CHISM scale, which was completed in a post-survey. A minimum interaction of three hours per week with each AIC, or 48 h over a month across all AICs, was requested from each participant.

The selection of the four AICs, namely Mondly, Andy, John Bot, and Buddy.ai, was guided by specific criteria, including multiplatform compatibility, wide availability, and diverse functionalities such as the integration of different technologies. These AICs offered a wide range of options, such as catering to different English language proficiency levels, providing personalized feedback, adapting to individual learning progress, and incorporating other technologies (AR, VR) in some cases. The aim was not to compare the four AICs, but rather to present teacher candidates with a broad overview of these virtual tutors, providing a variety of options and examples. Table 1 summarizes the main features of each AIC.

Concerning general perceptions of chatbots, the pre-survey results shown in Table  2 evidenced that participants found them most useful in providing general information (M = 3.4), while the perceived usefulness was lower for social media (M = 2.7), and education (M = 2.7). Analysis of the data indicated a uniform perspective among Spanish and Czech students on chatbots' role across general information, social media, and education. Mean scores in these areas were closely matched, highlighting a consensus on the significance of chatbots in these domains.

Regarding the frequency of use of the four AICs employed in the intervention, the post-survey results shown in Table  3 indicated that Andy was the most frequently used, averaging nearly 4 h per week, followed by John Bot and Mondly, while Buddy.ai was the least used. Concerning the educational setting, Spanish participants interacted more frequently with all four AICs compared to Czech students. The SD values show a similar level of variation in the weekly interaction hours across all four AICs for both Spanish and Czech participants, suggesting a comparable spread of interaction frequencies within each group.

Chatbot-human interaction satisfaction model results

The CHISM scale comprised three dimensions: Language Experience (LEX), Design Experience (DEX) and User Experience (UEX). Table 4 shows the results of the first dimension (LEX) aimed at measuring nine language-related features of the four AICs.

Semantic Coherent Behaviour (#1SCB) refers to the chatbot’s capacity to sustain a contextually relevant and meaningful dialogue with the user. Prior research has evidenced that student dissatisfaction with chatbot interaction is mainly due to the use of prearranged responses and off-topic remarks (Fryer et al., 2020 ; Kukulska-Hulme & Lee, 2020 ). Therefore, it is crucial for a chatbot to comprehend the conversation's context, provide appropriate responses, and recall past interactions. None of the AICs reached the moderate point of 3 on a five-point Likert scale, with Andy scoring the highest (M = 2.8), and closely followed by John Bot (M = 2.7). These scores can be explained by considering each AIC’s design and target audience. For instance, Mondly heavily depends on pre-programmed responses as ‘it is targeted at lower levels’ (Hajizadeh et al., 2023 : 12), requiring students to select a given response from a limited set of options. As a result, participants deemed the interaction as repetitive, because ‘if learners do not follow the assumed conversation patterns, the chatbot repeats the same questions until the learner provides the expected answer’ (Jung, 2019 : 77). Similarly, Buddy.ai is primarily designed for children aged 4–10 with a focus on oral skills through repetition drills. While these iterative approaches can ensure response accuracy and consistency at lower levels, they may restrict the chatbot's capability to engage in more dynamic and contextually relevant conversations, as pointed out in previous research (Gokturk, 2017 ).

Sentence Length and Complexity (#2SLC) pertains to the structure and variety of the sentences that a chatbot uses in its responses. Ideally, an AIC playing the role of a virtual tutor should adjust the length of its responses based on the learner's level, inputs and the context of the conversation, including the use of different grammatical constructs and vocabulary. Conversely, an AIC that only uses simple, repetitive patterns might come off as robotic or limited. Among the four AICs, Andy was perceived as the most proficient in varying sentence length and complexity depending on the learner’s inputs (M = 3.0). Unlike Mondly and Buddy.ai where the chatbots’ structure is limited according to the students’ lower levels, ‘Andy bot commands a high level of vocabulary and sentence structures and can deal with difficult topics; thus it can accommodate advanced learners with more free conversation style’ (Jung, 2019 : 79). As Andy is a conversation-oriented rather than a lesson-oriented chatbot, it can cater to different proficiency levels, from beginners to advanced, so it dynamically adapts to the individual language level of each learner.

The third item responds to Speech Recognition and Synthesis (#3R&S), which has received limited attention in relevant research (Jeon et al., 2023 ). Early chatbots relied on simple text-based inputs and outputs, and their ability to understand and respond to user questions was quite limited. Recognition technology, often referred to as Automatic Speech Recognition (ASR), converts spoken language into written text while Synthesis technology, also known as Text-to-Speech (TTS), converts written text into spoken words. R&S technology allows AICs to provide text and verbal responses to user inputs, making the interaction more engaging and human-like. Buddy.ai, with its emphasis on oral interaction for children, outperformed the other chatbots in this feature as the results in Table  4 show, while some users reported speech recognition problems with the other AICs, necessitating multiple repetitive attempts at interaction. These technical issues, partly due to an intentional accented pronunciation, made some participants feel anxious about their utterances and lowered their motivation to interact with the chatbots, in line with previous findings (Jeon, 2021 ). Additionally, the audio quality of certain AICs was criticized for their ‘robotic’ sound, as illustrated in the qualitative results presented in “ Teacher candidates’ perceptions of App-Integrated Chatbots ” section (Table  7 ). While R&S technology has come a long way, there are still several challenges that need to be addressed to improve the quality of chatbot-human interactions: accurate speech recognition, emotion recognition (intonation, pitch, rhythm), and natural language generation.

Items 4 and 5 are associated with the Quality of Vocabulary Reference (#4QVR) and Quality of Grammar Explanations (#5QGE) respectively. These items, which have been incorporated into the original CHISM scale to be utilized specifically with AICs, are essential for learners to comprehend word usage in context (synonyms, collocations) and understand grammar rules. Among the four AICs, Andy received the highest scores in both #4QVR (M = 3.1) and #5QGE (M = 2.9), indicating its higher effectiveness in providing comprehensive vocabulary and grammar references, closely followed by John Bot (#4QVR M = 3.0 and #5QGE M = 2.8). Both chatbots prioritize context-based explanations, offering learners the flexibility to personalize their learning progress (adaptive learning) in real time and engage with vocabulary and grammar rules in different contexts, as Fig.  2 illustrates. Furthermore, the integration of vocabulary and grammar exercises as reinforcement to the provided explanations played a crucial role in more extensive human-chatbot interaction, in line with previous findings (Jung, 2019 ).

figure 2

Screenshots of the participants’ interaction with Andy (left) and John Bot (right)

Error detection and correction (#6EDC) is another essential component of AICs as it plays a significant role in language learning. It enables the chatbot to identify and provide feedback on language errors in real time, helping learners improve their accuracy while fostering a supportive learning environment. Prior research has indicated that the use of AICs can reduce language anxiety, especially among less self-confident learners, by creating a non-judgmental environment where learners feel less inhibited to make errors and participate in conversations (Bao, 2019 ; Huang et al., 2022 ). However, despite previous findings highlighting the positive use of John Bot in this aspect (Khang et al., 2023 ), the EDC results shown in Table  4 indicated that none of the AICs achieved a moderate level of satisfaction. The scores were slightly higher for Andy (M = 2.4) and John Bot (M = −2.4), though. This can be attributed to two factors: first, the limited range of response options given in the interaction with some AICs, which restricted their ability to effectively identify and correct errors in a more natural conversation, as already pointed out by Jung ( 2019 ); and second, the lack of meaningful explanations provided by certain AICs, particularly those that are more lesson-oriented. Participants noted that Mondly and Buddy.ai lacked clear explanations to help learners understand and correct their errors. As pointed out by teacher candidates, the practice of merely asking for repetition of the same answers without further explanation could hinder the learning process, suggesting a need for improvements in this area.

Natural Conversational Interaction (#7NCI) pertains to the chatbot’s ability to emulate the natural flow and dynamics of human conversation. It involves several key elements, such as maintaining a contextually relevant conversation, understanding and responding appropriately to user inputs, demonstrating empathy, and adapting the language style and tone to suit the learner's preferences. The goal is to create a conversation that not only provides informative and accurate responses but also engages users in a manner that simulates a human-to-human interaction. None of the AICs reached the desired level of conversational naturalness, as participants found their responses predictable and lacking the adaptability seen in human tutors. As observed in previous research (Kukulska-Hulme & Lee, 2020 ), the limited range of response options and the inability of AICs to provide personalized feedback and multi-user interaction like a human tutor contributed to the perception that AICs fall short in simulating human tutoring experience.

Chatbot Response Interval (#8CRI) relates to the time it takes for a chatbot to generate a response following text or voice-based input from the learner. During chatbot interactions, there can be a delay in response due to factors such as processing time and system limitations as AICs often need additional time to analyse input, retrieve information, and generate suitable responses. The chatbot response time, whether overly rapid or delayed, can shape the overall dialogue experience and affect how engaged the learner is and how human-like they perceive the chatbot to be (Gnewuch et al., 2022 ). Several teacher candidates reported technical issues, particularly with the voice-enabled features of some AICs, as certain responses took longer or were not accurately processed. Among the four AICs, Andy demonstrated a higher score in #8CRI (M = 3.4), indicating a prompt and adaptive response to user inputs while Buddy.ai faced challenges in this aspect, particularly related to speech recognition problems, resulting in a lower score (M = 2.7).

Non-Verbal Language (#9NVL) in written communication refers to the use of visual cues, gestures, and symbols to convey meaning and emotions in the absence of spoken words. Emojis, memes, stickers, GIFs, and other visual representations are fundamental in social media and messaging apps today, yet their application in chatbots is an area that remains under-researched. These elements enable individuals to communicate more effectively, evoke specific emotions, and enhance the overall user experience (Beattie et al., 2020 ). Furthermore, NVL compensates for the limitations of text-based communication by filling the gap left by the absence of tone of voice, facial expressions, and body language. In the context of AICs, the integration of NVL can foster more engaging and expressive conversations, nurture a sense of community, and promote better understanding among learners. Based on the data, the teacher candidates perceived all AICs to have limited effectiveness in using visual cues and symbols to enhance communication and convey emotions, resulting in low results in NVL except for Andy (M = 2.9), which scored higher due to its more frequent use of these elements as illustrated in Fig.  2 .

The second dimension of the CHISM model, focusing on the Design Experience (DEX), underscores its critical role in fostering user engagement and satisfaction beyond the linguistic dimension. Elements such as the chatbot interface and multimedia content hold substantial importance in this regard. An intuitive and user-friendly interface enriches the overall user experience and encourages interaction (Chocarro et al., 2021; Yang, 2022 ). Additionally, the incorporation of engaging multimedia content, including videos, images, and other emerging technologies, can also increase users' attention and engagement (Jang et al., 2021 ; Kim et al., 2019 ). Table 5 shows the results of the three items included in the DEX dimension.

The Multimedia Content design (MC #10) encompasses the use of images and videos to create a multimedia-rich environment that enhances language learning (Zhang et al., 2023 ). Furthermore, AICs can be integrated with social media platforms, promoting collaboration and cultural exchange (Haristiani & Rifa’i, 2020 ). The integration of emerging technologies such as VR and AR further enhances the language learning experience, providing immersive and authentic learning environments. The results of the four AICs indicated that Buddy.ai achieved the highest score (M = 3.5), closely followed by Mondly (M = 3.1). The teacher candidates enjoyed Buddy.ai's innovative design as it specifically caters to children and incorporates mixed reality elements, so it creates a more engaging and immersive learning environment for young learners as Fig.  2 illustrates. Similarly, Mondly's AR component was praised as innovative, offering interactive lessons and realistic conversations for vocabulary learning and pronunciation practice although some participants considered AR more of a novelty than an effective learning tool.

The Game-Based Learning (#11GBL) component refers to the incorporation of challenges, rewards, progress bars, status indicators, and other game-based elements, to enhance learner motivation, track progress, and provide valuable feedback (Jung, 2019 ; Petrović & Jovanović, 2021 ). Based on the data, all four AICs received relatively positive results in terms of GBL with the highest score for Buddy.ai (M = 3.6), indicating its strong performance in incorporating game-based elements for children while talking with an interactive virtual tutor as Fig.  3 illustrates.

figure 3

Screenshots of Mondly AR (left) and Buddy.ai (right)

A Chatbot User Interface (#9CUI) that is intuitive, scalable, and provides easy access to a variety of multimedia materials may determine its success among language learners (Chocarro et al., 2023 ; Kim et al., 2019 ). Some chatbots even offer customization based on the user's profile, learning objectives, and preferences (Belda-Medina et al., 2022 ). In this sense, all AICs garnered positive feedback from teacher candidates, with Buddy.ai again achieving the highest average rating (M = 3.6), making it well-suited for children. As expressed by several teacher candidates, the importance of a dynamic and customizable CUI is on par with the linguistic abilities of the chatbot, as it affects their engagement and motivation. The results for each of the three items comprising the third CHISM dimension, User Experience (UEX), are presented in Table  6 .

The User Engagement (#13UENG) process involves several stages: initiation of interaction by learners, active involvement in tasks or actions, pausing or stopping which indicates disengagement, and re-engagement when learners return to the activity. In order to sustain learner engagement, AICs need to offer memory capacity, immediate assistance, customized experiences, and scalable options. Despite prior research highlighting the positive results in the use of Mondly (Hajizadeh et al., 2023 ) and John Bot (Khang et al., 2023 ), teacher candidates reported varying degrees of engagement with the four AICs, with the overall feedback not being as positive as expected. Both Mondly (M = 2.4) and John Bot (M = 2.5) were perceived as repetitive, while Andy (M = 2.8) was rated more favourably due to its memory capacity, adaptability, and extensive vocabulary range. Participants believed these characteristics brought Andy closer to a virtual intelligent tutor.

User Enjoyment (#14UENJ) pertains to the level of pleasure and satisfaction experienced by learners while interacting with the AIC (Chocarro et al., 2021; Yang, 2022 ). Learners' satisfaction can be shaped by engaging gamified activities, interactive conversational dialogues, visually appealing designs, motivating progress indicators, and immersive audio effects that collectively contribute to an enjoyable and enriching learning environment. However, the reported satisfaction levels were not as positive as expected in line with the #14UENG results. Mondly received the lowest score (M = 2.4), while Andy (M = 2.9) and Buddy.ai (M = 2.9) attained a moderate level of enjoyment, indicating that their incorporation of game elements and visually captivating aspects contributed to a more satisfying experience among teacher candidates.

User Further Interest (UFI #15) can be defined as the level of curiosity and eagerness displayed by students to explore and learn more about the AIC. It reflects users' willingness to delve deeper into the AIC's capabilities, engage with new functionalities, and continue using it in the future. UFI may be influenced by factors such as perceived value, continuous learning opportunities, and the incorporation of user feedback and adaptability. Users are more likely to be further interested in an AIC that they perceive as valuable and provides on-going learning opportunities (Fryer et al., 2019 ). The results obtained for Andy (M = 2.8) and Buddy.ai (M = 2.8) were moderate, while Mondly (M = 2.4) and John Bot (M = 2.3) received lower scores, aligning with the findings of the other two components of the UEX dimension.

Teacher candidates’ perceptions of app-integrated chatbots

Qualitative data, obtained from in-class discussions and assessment reports submitted through the Moodle platform, were systematically coded and categorized using QDA Miner. The goal was to analyse and identify the main benefits and drawbacks of each AIC as perceived by teacher candidates. These themes were cross-referenced with the different components of the CHISM model to establish correlations as shown in Table  7 . Frequency in the table refers to the number of observations made in the sample of textual data based on the written assessments provided by participants.

In line with previous research, the main advantages of AICs as perceived by participants were providing language support and delivering feedback on the learning progress via conversational interaction, in line with previous findings (Jeon, 2021 ; Yang, 2022 ), as well as enhancing user involvement through multimedia and interactive activities (Chuah & Kabilan, 2021 ; Dokukina & Gumanova, 2020 ). Conversely, the key limitations highlighted were their lack of adaptivity to varying proficiency levels (Huang et al., 2022 ), a tendency of some AICs to provide unrelated responses due to an over-reliance on predetermined answers (Huang et al., 2022 ), and notably, the imperfections in voice recognition capabilities (Bao, 2019 ). Table 7 provides a summary of the primary advantages and drawbacks of each AIC, along with their correlation to the items in the CHISM model, which are indicated in parentheses.

The CHISM results, particularly in the Language Experience (LEX) dimension, revealed significant insights about the teacher candidates’ perceptions of the four evaluated chatbots. When examining why none of the AICs achieved moderate satisfaction in the LEX dimension, it is crucial to consider each AIC's design and target audience limitations, as pointed out in previous research (Gokturk, 2017 ; Hajizadeh, 2023 ). For instance, Mondly's reliance on pre-programmed responses and Buddy.ai's focus on repetitive drills for children limit dynamic conversation, resulting in lower satisfaction in maintaining contextually relevant dialogues. Although Andy scores slightly higher, it still reveals a need for more adaptable conversation styles for advanced learners. The satisfaction levels in the LEX dimension may also depend on the chatbots' design relative to students' levels, with significant differences observed among the four AICs. For example, while Buddy.ai is oriented towards developing oral skills in children at a lower level, John Bot and Andy are designed for vocabulary and grammar building through role-playing interactions at more intermediate levels.

Additionally, speech technologies emerged as an area requiring substantial improvement, in line with previous results (Jeon et al., 2023 ). With the exception of Buddy.ai, the voice-based interactions provided very low results due to poor speech recognition and dissatisfaction with the synthesized voice, potentially leading to student anxiety and disengagement. Improvement could be achieved by investing in advanced technology capable of understanding a wide range of accents and mitigating background noise interference, possibly employing machine learning algorithms trained on various accents and speech patterns (Kohnke, 2023 ; Kukulska-Hulme & Lee, 2020 ). Simultaneously, rendering the AICs' voice generation more human-like can be attained through more sophisticated Text-to-Speech (TTS) systems that mimic the intonation, rhythm, and stress of natural speech (Jeon et al., 2023 ).

The findings indicate other key potential areas for AIC improvement to better cater to users' proficiency levels. It would be beneficial to implement more sophisticated AI algorithms capable of effectively assessing a user's language skills based on their real-time input and adjusting the chatbot's responses accordingly, as learners’ interest in chatbots is usually mediated by competence level. The development of LLM-power chatbots could help avoid irrelevant responses often resulting from an over-reliance on pre-set answers, as indicated by Jeon ( 2021 ).

Expanding on the necessity for improved customization in AICs, the integration of different features can be proposed to enhance chatbot-human personalization (Belda-Medina et al., 2022 ). These features include the ability to customize avatars (age, gender, voice, etc.) similar to intelligent conversational agents such as Replika. For example, incorporating familiar characters from cartoons or video games into chatbots can enhance engagement, particularly for children who are learning English by interacting with their favorite characters. Furthermore, by incorporating Augmented Reality (AR) technology, avatars can be launched and video calls can be enabled on social platforms such as Kuki.ai, thereby adding a layer of personal interaction. Looking ahead, allowing students to select specific design aspects of AICs, similar to choosing linguistic features such as target level or accent, could be a crucial step in creating a more adaptive and personalized learning experience.

Conclusion and implications

The application of the CHISM model in the evaluation of four AICs has provided valuable insights into the effectiveness of these tools in language learning. The model, which comprises three dimensions (LEX, DEX, UEX), has allowed for a comprehensive assessment of the AICs across multiple facets. The Language Experience dimension (LEX), which includes elements such as Semantic Coherent Behaviour, Sentence Length and Complexity, and Speech Recognition and Synthesis, revealed that none of the AICs reached the moderate point of satisfaction among EFL teacher candidates. This suggests that while these tools have made strides in providing language-related features, there is still room for improvement, particularly in terms of maintaining contextually relevant dialogues and varying sentence complexity based on the learner's level.

The Design Experience dimension (DEX) underscored the importance of user-friendly interfaces and engaging multimedia content in fostering user engagement and satisfaction. The findings uncovered the necessity for enhancements in adaptive user interfaces, as well as the incorporation of social media and emerging technologies, to simulate the human-student interaction and enrich the language learning experience. The User Experience dimension (UEX) revealed that while some AICs were able to provide a moderate level of enjoyment and engagement, overall satisfaction levels were not as positive as expected. This indicates the need for AICs to offer a more personalized learning experience to sustain learner engagement and interest.

The CHISM model offers a comprehensive approach to evaluating AICs, encompassing not only linguistic capabilities but also design and user experience aspects. This holistic evaluation allows for a more nuanced understanding of the strengths and weaknesses of AICs, providing valuable insights for future improvements. The model also highlights the potential of AICs in language learning, particularly in terms of providing immediate feedback, and fostering a supportive learning environment.

However, the study also highlights the challenges that need to be addressed, such as the requirement for more sophisticated AI algorithms capable of adjusting to the learner's proficiency level and the improvement of speech technologies. This suggests the need for evolving teaching methods and curricula to more effectively incorporate AICs, emphasizing the enhancement of their capabilities for providing contextually rich and varied linguistic experiences. One practical approach could be the introduction of specific learning modules on different types of chatbots, such as app-integrated, web-based, and standalone tools, as well as Artificial Intelligence, into the curriculum. Such modules would equip students and future educators with a deeper understanding of these technologies and how they can be utilized in language education. The implications of these findings are significant, as they provide a roadmap for the development of more effective and engaging AICs for language learning in the future.

Limits and future directions

This study has three main limitations: firstly, the cross-sectional design and reliance on self-reported data may limit the ability to establish causality; secondly, the specific context and non-probabilistic method may restrict the generalizability of findings, necessitating replication in different settings and populations; thirdly, the focus on four specific AICs may not fully capture the complexity of this new technology, underscoring the need to incorporate a broader range of AICs for a more comprehensive evaluation. Further research in the field of AICs could encompass a variety of areas, such as investigating their effect with different language levels, as well as the efficacy of AICs across different student age groups and contexts. Finally, the impact of various chatbot design elements on student interaction and engagement in language learning could be explored. These elements could include multimedia integration, the incorporation of social media, and notably, the use of speech technologies.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due privacy reasons but are available from the corresponding author on reasonable request.

Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2 , 100006. https://doi.org/10.1016/j.mlwa.2020.100006

Article   Google Scholar  

Ajisoko, P. (2020). The use of Duolingo apps to improve English vocabulary learning. International Journal of Emerging Technologies in Learning (iJET), 15 (7), 149–155. https://doi.org/10.3991/ijet.v15i07.13229

Annamalai, N., Eltahir, M. E., Zyoud, S. H., Soundrarajan, D., Zakarneh, B., & Al Salhi, N. R. (2023). Exploring English language learning via Chabot: A case study from a self determination theory perspective. Computers and Education: Artificial Intelligence . https://doi.org/10.1016/j.caeai.2023.100148

Ayedoun, E., Hayashi, Y., & Seta, K. (2015). A conversational agent to encourage willingness to communicate in the context of English as a foreign language. Procedia Computer Science, 60 , 1433–1442. https://doi.org/10.1016/j.procs.2015.08.219

Bao, M. (2019). Can home use of speech-enabled artificial intelligence mitigate foreign language anxiety–investigation of a concept. Arab World English Journal (AWEJ), 5 , 28–40. https://doi.org/10.24093/awej/call5.3

Beattie, A., Edwards, A. P., & Edwards, C. (2020). A bot and a smile: Interpersonal impressions of chatbots and humans using emoji in computer-mediated communication. Communication Studies, 71 (3), 409–427. https://doi.org/10.1080/10510974.2020.1725082

Belda-Medina, J., & Calvo-Ferrer, J. R. (2022). Using chatbots as AI conversational partners in language learning. Applied Sciences, 12 (17), 8427.

Chen, H.-L., Vicki Widarso, G., & Sutrisno, H. (2020). A chatbot for learning Chinese: Learning achievement and technology acceptance. Journal of Educational Computing Research, 58 (6), 1161–1189. https://doi.org/10.1177/073563312092

Chocarro, R., Cortiñas, M., & Marcos-Matás, G. (2023). Teachers’ attitudes towards chatbots in education: A technology acceptance model approach considering the effect of social language, bot proactiveness, and users’ characteristics. Educational Studies . https://doi.org/10.1080/03055698.2020.1850426

Chuah, K.-M., & Kabilan, M. (2021). Teachers’ views on the use of Chatbots to support English language teaching in a mobile environment. International Journal of Emerging Technologies in Learning (iJET), 16 (20), 223–237. https://doi.org/10.3991/ijet.v16i20.24917

Dokukina, I., & Gumanova, J. (2020). The rise of chatbots–new personal assistants in foreign language learning. Procedia Computer Science, 169 , 542–546. https://doi.org/10.1016/j.procs.2020.02.212

El Shazly, R. (2021). Effects of artificial intelligence on English speaking anxiety and speaking performance: A case study. Expert Systems, 38 (3), e12667. https://doi.org/10.1111/exsy.12667

Fryer, L., Coniam, D., Carpenter, R., & Lăpușneanu, D. (2020). Bots for language learning now: Current and future directions. Language Learning & Technology, 24 (2), 8–22.

Google Scholar  

Fryer, L., Nakao, K., & Thompson, A. (2019). Chatbot learning partners: Connecting learning experiences, interest and competence. Computers in Human Behavior, 93 , 279–289. https://doi.org/10.1016/j.chb.2018.12.023

Gnewuch, U., Morana, S., Adam, M. T., & Maedche, A. (2022). Opposing effects of response time in human-chatbot interaction: The moderating role of prior experience. Business & Information Systems Engineering, 64 (6), 773–791. https://doi.org/10.1007/s12599-022-00755-x

Godwin-Jones, R. (2022). Chatbots in language learning: AI systems on the rise. In B. Arnbjörnsdóttir, B. Bédi, L. Bradley, K. Friðriksdóttir, H. Garðarsdóttir, S. Thouësny, & M. J. Whelpton (Eds.), Intelligent CALL, granular systems and learner data: short papers from EUROCALL (pp. 124–128). Research-publishing.net. https://doi.org/10.14705/rpnet.2022.61.1446

Chapter   Google Scholar  

Gokturk, N. (2017). Mondly learning languages. Pronunciation in Second Language Learning and Teaching Proceedings, 8 (1), 241–247.

Hajizadeh, S., Salman, A. R., & Ebadi, S. (2023). Evaluating language learning applications from efl learners’ perspectives: The case of mondly. Research Square . https://doi.org/10.21203/rs.3.rs-3011332/v1

Hakim, R., & Rima, R. (2022). Chatting with AI Chatbots applications to improve English communication skill. Journal of English Language Studies., 7 (1), 121–130. https://doi.org/10.30870/jels.v7i1.14327

Haristiani, N., Danuwijaya, A. A., Rifai, M. M., & Sarila, H. (2019). Gengobot: A chatbot-based grammar application on mobile instant messaging as language learning medium. Journal of Engineering Science and Technology, 14 (6), 3158–3173.

Haristiani, N., & Rifa’i, M. M. (2020). Combining chatbot and social media: Enhancing personal learning environment (PLE) in language learning. Indonesian Journal of Science and Technology, 5 (3), 487–506. https://doi.org/10.17509/ijost.v5i3.28687

Hsu, M.-H., Chen, P.-S., & Yu, C.-S. (2021). Proposing a task-oriented chatbot system for EFL learners speaking practice. Interactive Learning Environments . https://doi.org/10.1080/10494820.2021.1960864

Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38 (1), 237–257. https://doi.org/10.1111/jcal.12610

Hwang, G.-J., & Chang, C.-Y. (2021). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 31 (7), 4099–4112. https://doi.org/10.1080/10494820.2021.1952615

Jang, J., Ko, Y., Shin, W. S., & Han, I. (2021). Augmented reality and virtual reality for learning: An examination using an extended technology acceptance model. IEEE Access, 9 , 6798–6809. https://doi.org/10.1109/ACCESS.2020.3048708

Jeon, J. (2021). Exploring AI chatbot affordances in the EFL classroom: Young learners’ experiences and perspectives. Computer Assisted Language Learning . https://doi.org/10.1080/09588221.2021.2021241

Jeon, J., Lee, S., & Choi, S. (2023). A systematic review of research on speech-recognition chatbots for language learning: Implications for future directions in the era of large language models. Interactive Learning Environments . https://doi.org/10.1080/10494820.2023.2204343

Jung, S. K. (2019). Introduction to popular mobile chatbot platforms for English learning: Trends and issues. STEM Journal, 20 (2), 67–90. https://doi.org/10.16875/stem.2019.20.2.67

Article   MathSciNet   Google Scholar  

Khang, A., Muthmainnah, M., Seraj, P. M. I., Al Yakin, A., & Obaid, A. J. (2023). AI-Aided teaching model in education 5.0. In A. Khang, V. Shah, & S. Rani (Eds.), Handbook of research on AI-based technologies and applications in the Era of the metaverse (pp. 83–104). IGI Global.

Kharis, M., Schön, S., Hidayat, E., Ardiansyah, R., & Ebner, M. (2022). Mobile Gramabot: Development of a chatbot app for interactive German grammar learning. International Journal of Emerging Technologies in Learning, 17 (14), 52–63. https://doi.org/10.3991/ijet.v17i14.31323

Kim, N.-Y. (2019). A study on the use of artificial intelligence chatbots for improving English grammar skills. Journal of Digital Convergence, 17 (8), 37–46. https://doi.org/10.14400/JDC.2019.17.8.037

Kim, Nao-Young. (2020). Chatbots and language learning: effects of the use of AI chatbots for EFL learning . Eliva press.

Kim, N.-Y., Cha, Y., & Kim, H.-S. (2019). Future English learning: Chatbots and artificial intelligence. Multimedia-Assisted Language Learning, 22 (3), 32–53.

Kohnke, L. (2023). L2 learners’ perceptions of a chatbot as a potential independent language learning tool. International Journal of Mobile Learning and Organisation, 17 (1–2), 214–226. https://doi.org/10.1504/IJMLO.2023.10053355

Kukulska-Hulme, A., & Lee, H. (2020). Intelligent assistants in language learning: An analysis of features and limitations. In K.-M. Frederiksen, S. Larsen, L. Bradley, & S. Thouësny (Eds.), CALL for widening participation: Short papers from EUROCALL (pp. 172–176). Research-publishing.net.

Li, Y., Chen, C.-Y., Yu, D., Davidson, S., Hou, R., Yuan, X., Tan, Y., Pham, D., & Yu, Z. (2022). Using Chatbots to Teach Languages. Proceedings of the Ninth ACM Conference on Learning@ Scale , 451–455. https://doi.org/10.1145/3491140.3528329

Nastasi, B. K., Hitchcock, J., Sarkar, S., Burkholder, G., Varjas, K., & Jayasena, A. (2007). Mixed methods in intervention research: Theory to adaptation. Journal of Mixed Methods Research, 1 (2), 164–182. https://doi.org/10.1177/155868980629

Panesar, K. (2020). Conversational artificial intelligence-demystifying statistical vs linguistic NLP solutions. Journal of Computer-Assisted Linguistic Research, 4 , 47–79. https://doi.org/10.4995/jclr.2020.12932

Petrović, J., & Jovanović, M. (2021). The role of chatbots in foreign language learning: The present situation and the future outlook. Artificial Intelligence: Theory and Applications, 2 , 313–330. https://doi.org/10.1007/978-3-030-72711-6_17

Pham, X. L., Pham, T., Nguyen, Q. M., Nguyen, T. H., & Cao, T. T. H. (2018). Chatbot as an intelligent personal assistant for mobile language learning. Proceedings of the 2018 2nd International Conference on Education and E-Learning , 16–21. https://doi.org/10.1145/3291078.3291115

Pin-Chuan Lin, M., & Chang, D. (2020). Enhancing post-secondary writers’ writing skills with a chatbot: A mixed-method classroom study. Journal of Educational Technology & Society, 23 (1), 78–92.

Rapp, A., Curti, L., & Boldi, A. (2021). The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies, 151 , 102630. https://doi.org/10.1016/j.ijhcs.2021.102630

Skjuve, M., Følstad, A., Fostervold, K. I., & Brandtzaeg, P. B. (2021). My chatbot companion-a study of human-chatbot relationships. International Journal of Human-Computer Studies, 149 , 102601. https://doi.org/10.1016/j.ijhcs.2021.102601

Vera, F. (2023). Integrating Artificial Intelligence (AI) in the EFL classroom: Benefits and challenges. Transformar, 4 (2), 66–77.

Yang, J. (2022). Perceptions of preservice teachers on AI chatbots in English education. International Journal of Internet, Broadcasting and Communication, 14 (1), 44–52. https://doi.org/10.7236/IJIBC.2022.14.1.44

Zhang, R., Zou, D., & Cheng, G. (2023). A review of chatbot-assisted learning: Pedagogical approaches, implementations, factors leading to effectiveness, theories, and future directions. Interactive Learning Environments . https://doi.org/10.1080/10494820.2023.2202704

Download references

Acknowledgements

The authors would like to express their gratitude to all the college students from both institutions for their invaluable participation in this project.

Not applicable.

Author information

Authors and affiliations.

Department Filología Inglesa, Facultad de Letras, Campus de San Vicente, University of Alicante, 03690, San Vicente, Alicante, Spain

Jose Belda-Medina

Institute of Applied Language Studies, University of West Bohemia, Plzen, Czech Republic

Vendula Kokošková

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization was done by both JB and VK. The methodology was developed by JB. Both JB and VK were responsible for the software and treatment. Validation and formal analysis were carried out by both JB and VK. The original draft was written by JB and reviewed and edited by VK. Both authors have read and agreed to the published version of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jose Belda-Medina .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Belda-Medina, J., Kokošková, V. Integrating chatbots in education: insights from the Chatbot-Human Interaction Satisfaction Model (CHISM). Int J Educ Technol High Educ 20 , 62 (2023). https://doi.org/10.1186/s41239-023-00432-3

Download citation

Received : 19 September 2023

Accepted : 11 December 2023

Published : 19 December 2023

DOI : https://doi.org/10.1186/s41239-023-00432-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Language learning
  • Teacher candidates

designing an educational chatbot a case study of cikguaibot

  • DOI: 10.48550/arXiv.2306.10645
  • Corpus ID: 259202915

Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy

  • Cansu Koyuturk , Mona Yavari , +9 authors D. Ognibene
  • Published in arXiv.org 18 June 2023
  • Education, Computer Science

Figures from this paper

figure 1

8 Citations

An experiment on leveraging chatgpt for online teaching and assessment of database students, llms of catan: exploring pragmatic capabilities of generative chatbots through prediction and classification of dialogue acts in boardgames’ multi-party dialogues, chatgpt based chatbot application, ai tutor enhanced with prompt engineering and deep knowledge tracing, learning to prompt in the classroom to understand ai limits: a pilot study, large language models can help boost food production, but be mindful of their risks, user modeling in the era of large language models: current research and future directions, artificial intelligence and social media on academic performance and mental well-being: student perceptions of positive impact in the age of smart learning, 52 references, interacting with educational chatbots: a systematic review, rediscovering the use of chatbots in education: a systematic literature review, large language models in education: a focus on the complementary relationship between human teachers and chatgpt, a categorical archive of chatgpt failures.

  • Highly Influential

Media Literacy Education in the Age of Machine Learning

Promptmaker: prompt-based prototyping with large language models, co-writing screenplays and theatre scripts with language models: evaluation by industry professionals, living up to the chatbot hype: the influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions, conversation style transfer using few-shot learning, the pedagogical challenges to collaborative technologies, related papers.

Showing 1 through 3 of 0 Related Papers

Computer Science > Human-Computer Interaction

Title: developing effective educational chatbots with chatgpt prompts: insights from preliminary tests in a case study on social media literacy (with appendix).

Abstract: Educational chatbots come with a promise of interactive and personalized learning experiences, yet their development has been limited by the restricted free interaction capabilities of available platforms and the difficulty of encoding knowledge in a suitable format. Recent advances in language learning models with zero-shot learning capabilities, such as ChatGPT, suggest a new possibility for developing educational chatbots using a prompt-based approach. We present a case study with a simple system that enables mixed-turn chatbot interactions and discuss the insights and preliminary guidelines obtained from initial tests. We examine ChatGPT's ability to pursue multiple interconnected learning objectives, adapt the educational activity to users' characteristics, such as culture, age, and level of education, and its ability to use diverse educational strategies and conversational styles. Although the results are encouraging, challenges are posed by the limited history maintained for the conversation and the highly structured form of responses by ChatGPT, as well as their variability, which can lead to an unexpected switch of the chatbot's role from a teacher to a therapist. We provide some initial guidelines to address these issues and to facilitate the development of effective educational chatbots.
Comments: Poster version accepted at the 31st International Conference on Computers in Education (ICCE)
Subjects: Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
Cite as: [cs.HC]
  (or [cs.HC] for this version)
  Focus to learn more arXiv-issued DOI via DataCite

Submission history

Access paper:.

  • Other Formats

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

BibSonomy logo

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

  • Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

designing an educational chatbot a case study of cikguaibot

Design of a Chatbot Learning System: Case Study of a Reading Chatbot

' src=

Bulletin of the Technical Committee on Learning Technology (ISSN: 2306-0212)
Volume 22, Number 1, 2-7 (2022)
Received December 13, 2021
Accepted January 29, 2022
Published online February 21, 2022
This work is under Creative Commons CC-BY-NC 3.0 license. For more information, see

email

Technology is more than a tool; the use of technology is also a skill that needs to be developed. Teachers are expected to have the ability to integrate technology into their teaching methods, but whether they have the required technological expertise is often neglected. Therefore, technologies that can be easily used by teachers must be developed. In this study, an algorithm was developed that integrates Google Sheets with Line to offer teachers who are unfamiliar with programming a quick method for constructing a chatbot based on their teaching plan, question design, and the material prepared for a reading class. To meet the needs for reading classes, reading theories and effective reading instruction are incorporated into the learning and teaching mechanism of the chatbot system. To create a guidance structure that is suitable for students of various levels of ability, Nelson’s multipath digital reading model was employed because it can maintain a reading context while simultaneously responding to the diverse reading experiences of different readers.

Keywords: Educational technology, Learning management systems, Mobile learning

I. INTRODUCTION

According to [ 1 ], the use of technological tools to supplement teaching and learning activities can help students access information efficiently, develop self-directed learning in students, and improve the quality of instruction. However, many teachers continue to adhere to traditional teaching methods rather than integrating technology into teaching because of their negative attitude toward and low confidence in technology use; insufficient professional knowledge and technological competence; and a lack of technological resources, training, and support [ 2 ], [ 3 ]. Therefore, this study focused on the development of a system to cater to teachers’ technological abilities and teaching needs.

In this study, a reading class was considered the target context. In accordance with the theory of reading comprehension, Kintsch’s single-text [ 4 ] and Britt’s multitext [ 5 ] reading comprehension models were integrated into the learning mechanism design. To provide assistance to students on the basis of the level of comprehension with which they have difficulty, Nelson’s multipath reading model was employed to design the question and answer mechanism [ 6 ].

To make the system easily operable and accessible for teachers who lack a programming background, Line, which is the most used communication platform in Taiwan, was used as the front-end interface, and Google Sheets, a commonly used cloud-based spreadsheet, was employed as the database containing teaching content and learning records. Moreover, programs and algorithms were developed using Google App Script to connect the Line and Google Sheets services.

A. Models of Reading Comprehension

According to Kintsch’s reading comprehension model, which is called the construction–integration model, reading comprehension is a process of continuous construction and integration [ 4 ], [ 7 ]. In this model, each sentence in a text is transformed into a semantic unit, which is called a proposition. The reader then constructs a coherent understanding by continually recombining these propositions in an orderly fashion. Reference [ 8 ] reviewed studies on single-text processing and assumed that the reading process involves at least three levels of memory representation. The surface level represents decoding of word meaning in the early reading stage. The textbase level represents the process of transforming a text into a set of propositions on the basis of lexical knowledge, syntactic analysis, and information retrieved from memory. The situation level represents the process of constructing a coherent understanding of the situation described in the text through experience accumulated in life.

The main limitation of a single text is that it only reflects the viewpoint of a specific author rather than offering the comprehensive viewpoints. Even when arguments are objectively summarized in a literature review, the author still selects from among original sources.  According to [ 5 ] and [ 9 ], if students are to address an issue critically and know how to construct a complete understanding of an issue, they should be allowed to learn by reading actual texts, practice selecting and organizing information, and interpret thoughts in their own manner. In multitext reading, texts have the role of providing raw information; reader must thus be clear on the purpose to their reading if they are to select and integrate relevant information and manage diverse or even contradictory viewpoints; otherwise, they may become lost in the ocean of information. Britt et al. extended the Kintsch model to propose the documents model and suggested that a higher level of proposition is event related and includes several clauses and paragraphs; this level involves understanding construction in multitext reading [ 5 ], [ 10 ]. Reference [ 8 ] reviewed studies on multitext reading and concluded that the reading process involves at least three memory representations: the integrated model represents the reader’s global understanding of the situation described in several texts, the intertext model represents their understanding of the source material, and the task model represents their understanding of their goals and appropriate strategies for achieving these goals. Compared with Kintsch’s theory, multitext reading theory is more reader-directed and emphasizes the reader’s approach to constructing a coherent and reasonable understanding from texts representing various viewpoints.

As suggested in [ 8 ], the challenge faced in the teaching of multiple-document reading is how to design a guidance structure that considers the reading paths of different students. Nelson proposed a digital reading model that can maintain a context and simultaneously respond to the diverse reading experiences of different readers. Nelson suggested breaking a text into smaller units and inserting hyperlinks in these units, allowing readers to jump from the current document to the content pointed to by the hyperlinks without affecting the structure of the text [ 6 ]. Moreover, reference [ 11 ] used Nelson’s model in a clear manner by treating reading units as nodes, interunit relationships as links, and reading experience as a network composed of nodes and links. Therefore, the collection of content with which the reader interacts can be treated as a representation of the reader’s reading process. Nelson’s multipath digital reading model inspired us to shift the complex teacher–student interaction during reading instruction to a chatbot system. Learning content can be considered a node, and question–answer pairs can be considered links to related learning content. If question–answer pairs fully represent students’ understanding, the students can be guided to the content they require on the basis of the answer they select. The following section explains the factors that must be accounted for within a well-designed question–answer framework.

B. Design of Questions and Instructions

Two particular reading interventions are employed to promote comprehension: an instructional framework based on self-regulated learning targets, which is used for basic-level comprehension, and a framework based on teacher-facilitated discussion targets, which is employed for  high-level comprehension and critical–analytic thinking [ 12 ]. Among interventions for teacher-facilitated discussion, questioning differs from direct explanation and strategic interventions, which help students develop reading comprehension through direct transfer of skills. Instead, questioning, involving asking students questions in a step-by-step manner, helps them actively construct and develop their understanding of a text.

A good question does not always come to mind easily; thus, teachers must prepare well before class. According to [ 13 ], before designing questions, teachers must have a general understanding of the text, consider probable student reactions, possess specific thinking skills, and decide which competencies should be evaluated.  According to [ 14 ] and [ 15 ], when designing questions, the level of the question should be based on the complexity of the cognitive processing required to answer the question. For example, factual questions, requiring the lowest level of processing, require students to recall relevant information from the text; paraphrased questions require students to recall specific concepts and express them in their own way; interpretive questions require students to search for and deduce a relationship among concepts that are not explicitly stated in the text; and evaluative questions, requiring the highest level of processing, require students to analyze and evaluate a concept in the text by using the context and their prior knowledge.

Questions can not only reflect the level of comprehension but also promote thinking. If higher-level questions are posed, students are more likely to think beyond the surface of the topic [ 16 ]. For example, even if a student can answer factual questions correctly, they do not necessarily gain insight from the facts. If the teacher then asks the student to paraphrase or interpret a concept, which would indicate whether the student can link facts together, the student is likely to demonstrate higher-level competency [ 16 ].

In recent years, the OECD’s Programme for International Student Assessment reading comprehension standards [ 17 ] have increasingly emphasized the role of the reader’s personal reflection in reading comprehension. However, irrespective of whether the questions require the students to organize information from texts, use their prior knowledge, or reflect on their life experiences, students must respond in accordance with the context of the text. In other words, they should not express their opinions and feelings freely as they wish. If making deductions from a text is the main competency to be assessed, the level of students’ comprehension can be determined by evaluating their selection of original sources while expressing their thoughts. Moreover, if students are asked to cite original sources, they are more likely to avoid straying from the topic and to demonstrate critical thinking [ 9 ].

To create a good questioning practice, teachers must consider the different levels of the various students and provide assistance accordingly. The different types of questions represent different levels of reading comprehension. Higher-order questions involve more complex cognitive strategies than strategic lower-order questions. Reference [ 18 ] stated that for students who have trouble in constructing meaning from a text, teachers should provide a supporting task, such as word recognition. References [ 14 ] and [ 19 ] have highlighted that for students who need help answering challenging questions, teachers should encourage more advanced use of thinking skills, such as metacognition and awareness of thinking.

The instant feedback that a teacher can provide on the basis of a student’s reply cannot be easily replaced by a predetermined instructional framework. Instead of replacing face-to-face instruction in a class, the system aims to solve the problems encountered during oral question-and-answer sessions and to provide teachers with students’ learning information to enable further counseling. Because identifying how students make inferences from texts is difficult for a teacher during oral communication, a recording mechanism is needed to help the teacher note the source of a student’s inference. According to [ 20 ], even if a teacher is well prepared, poor oral presentation skills can affect students’ understanding of questions. Therefore, a digital tool that fully implements a teacher’s questioning framework can be used to prevent misunderstanding. According to [ 21 ], some students fail to take the opportunity to practice because they feel reluctant to express themselves in public; thus, an individual-oriented learning system can ensure that every student practices an equal amount.

By summarizing the aspects that needed to be considered in the design of questions and instructions, the main guidelines of this system were defined as follows. The question design should support true/false questions, multiple-choice questions, and essay questions for different levels of students. The mechanism of replying to a question should support self-expression and connection with corresponding resources. The system must provide a basic mechanism for determining students’ level of reading comprehension from their qualitative reply and guide them to reread the text for self-modification and self-monitoring.

C. Application of a Chatbot

The earliest chatbot—ELISA, developed by Weizenbaum in 1966—used algorithmic processing and predefined response content to interact with humans [ 22 ]. Chatbots are commonly used to assist individuals in completing specific tasks, and the dialogues are designed to be purposeful and guided [ 23 ].

Recently, chatbots have been widely applied in educational settings and have been demonstrated to have beneficial effects on learning. For example, in [ 24 ] and [ 25 ], chatbots were applied to language learning and determined to induce interest and motivation in learning and increase students’ willingness to express themselves.  The results of one study [ 26 ], in which chatbots were applied to computer education revealed that students who learned in the chatbot-based learning environment performed comparably to those who learned through traditional methods. Moreover, [ 27 ] recently developed a chatbot framework by using natural language processing (NLP) to generate appropriate responses to inputs. They used NLP to distinguish and label students’ learning difficulties, connect students with the corresponding grade-level learning subjects, and quickly search for learning content that met the students’ needs. Other scholars [ 28 ] applied a chatbot to the learning management system of a university and employed artificial intelligence to analyze the factors that motivate students to learn actively, monitor academic performance, and provide academic advice. The results indicated that the method improved student participation in their course.

Many commonly used communication platforms and free digital resources now support the development of chatbots. Designing and maintaining a system of teaching aids would be time-consuming.  Chatbots already have high usability and are accepted by the public, meaning that using an existing platform to develop a chatbot would reduce users’ cognitive load during the learning process. Therefore, this study developed algorithms to link the services of two open source platforms, Google and Line, and create a cloud spreadsheet that can act as a database for storing teaching content and learning records. Because the algorithms connect with a spreadsheet, creating a new chatbot learning system by using the proposed approach is easy; the spreadsheet would be duplicated, and the setting would be updated with information on the new chatbot.

II. DESIGN OF SYSTEM

A. instructional flow design, 1)   structure.

A piece of text contains several propositions, and the propositions may be parallel or subordinate to a large proposition. Therefore, the structure of textual analysis and the teaching structure are hierarchical. The proposed system has three levels: the text, chapter, and content levels (Fig. 1). Each card in a carousel template represents one text, and having multiple texts is acceptable (Fig. 2).  Chatbot designers can update the chatbot interface and carousel template on the basis of their teaching structure once they have added a new text in Google Sheets (Fig. 2). Students can select any text they want to learn from at any time because of a menu button, called “Classes for Guided Reading of Texts,” which prompts the carousel template to pop up (Fig. 3). Each chapter has its own ID number, and the system connects the chapter’s learning content by the ID. For example, the ID of “Characteristic” is “01”; thus, if students press the button showing “Characteristic”, the system searches for the teaching content labeled “010000” for publishing on the chatbot and then moves to the next content in accordance with the next ID assigned by the designer (Fig. 4).

designing an educational chatbot a case study of cikguaibot

Fig. 1. Teaching structure (for a sample text).

Fig. 2. carousel template..

designing an educational chatbot a case study of cikguaibot

Fig. 3. Rich menu.

designing an educational chatbot a case study of cikguaibot

Fig. 4. Teaching content.

2) instructional content design.

According to Kintsch’s theory, instructions should assist students on the basis of the level at which they fail to arrive at a correct understanding of the text. In the surface level, instructions should provide word explanations. In the textbase level, instructions should help connect propositions that the students have ignored. In the situation level, the system should guide students in expressing a concept in their own way and in accordance with their experience. In some cases, the coherence between instruction contents that are not distinct is strong. Therefore, the teacher’s instructional flow can be designed as a linear structure or created with branches and flexibility to help guide students to the content at an appropriate level depending on whether the student knows specific concepts.

Teaching content that comprises an instructional flow is coded. The content in question form can be used to create a branch for the instructional flow. Each question can accept up to 13 branches. To arrange the next content to be published, the system requires the teacher to assign IDs to the branches of each question. According to multitext reading theory, at the integrated level, instructions should guide students to construct a global understanding of the texts. Therefore, each content ID is generated uniquely so that the next ID to be assigned is not limited to the range of texts currently being learned. For paraphrased questions that require students to respond in their own way and when no answer accurately represents a student’s thoughts, the system allows the student to reply by typing out a response if the next ID is set to “000000” (Fig. 4). The system stores the student’s learning progress by recording the order in which the student encountered the content, the buttons they pressed, and their replies (Fig. 5).

For both multiple-choice and paraphrased questions, the system asks the student to provide their qualitative reasoning and original sources; their responses enabled us to understand how students interpret texts (details in section II-B-5). In the case of a student’s thought not being represented by any answer, the student’s qualitative reply is treated as an exceptional case not considered by the teacher during the design stage, and all such replies are collected and given to the teacher.

designing an educational chatbot a case study of cikguaibot

Fig. 5. Learning record.

B. design of question and answer mechanism, 1) questioning mechanism.

Whether the students answer a question correctly does not reflect whether they fully understand a text. Examining the process of students’ interpretation can be a way to accurately follow their real thinking. According to Kintsch’s construction–integration model, a text is a combination of multiple propositions. Similarly, a reading comprehension question must be answered by combining different propositions. Therefore, by comparing the combinations of propositions used by the teacher and the students, it can be determined whether students have overlooked specific aspects, and appropriate guiding questions can then be provided to help the students review the text.

2) Text Processing

To help the teacher more effectively identify the connection between student responses and the text, the system cuts and codes the text provided by teachers by using the punctuation mark as a breakpoint. The system then creates a webpage on the basis of these sentence units and gives students the link to the webpage in a chatbot dialog (Fig. 6). The webpage has a guide that helps students reply, explain their reasoning, and pick sentences as resources to support their viewpoint (Fig. 7). The webpage is connected to the Line Login service; thus, the user’s identity is recognized and students’ replies are recorded and sent back to the same Google Sheet for the chatbot system and another subsheet for storage (Fig. 8)

designing an educational chatbot a case study of cikguaibot

Fig. 6. Chatbot dialog when answering a question.

designing an educational chatbot a case study of cikguaibot

Fig. 7. The webpage.

designing an educational chatbot a case study of cikguaibot

Fig. 8. Record of the reply in Google Sheets.

3) sentence marker.

When a teacher designs questions, they usually have a reference answer in mind and need to refer to specific information and propositions from the text for support, interpretation, and inference. Therefore, teachers are asked to provide reference answers with the corresponding textual sources when designing questions. Similarly, students must select corresponding textual sentences as the basis for their interpretations. According to multitext reading theory, at the intertext level, sourcing across texts is one of the main competencies that must be developed and evaluated if each sentence is to be coded uniquely. Students can pick sentences across texts.

4) Sentence Match

To calculate the similarity between a student’s answer and the reference answer provided by the teacher, the system compares the references of both. On the basis of the difference between the references, the system can distinguish and label the completeness of the student’s comprehension and provide a guiding question with which the student can review the text.

5) Qualitative Replies Classification and Analysis

Because the learning patterns of a group of students are unknown at the beginning of a course, the teacher should track students’ learning process in the long term and observe how students’ explanations and sentence selection evolve under the influence of the guiding questions provided by the system. Before analysis, if a user’s replies include multiple-choice selections and qualitative explanations with supporting sentences, the replies are classified into correct and incorrect. If a user’s replies are paraphrased replies rather than multiple-choice selections, their correctness is determined manually because the system is not yet capable of automatically determining correctness. Another area of analysis in which we are interested is comparing how different students interpret a given question; thus, we plan to classify qualitative explanations on the basis of sentence IDs.

III. SUMMARY AND FUTURE RESEARCH

The integration of technology into teaching requires consideration of many aspects, such as the teacher’s attitude, teacher’s technological knowledge and ability, and teaching needs, which are often overlooked. Because we believe that tools should be useful, not just usable, this study aimed to develop a teacher-friendly teaching-aid system based on theories of the teaching and learning of reading and empirical studies of technology applications.

Thanks to the advancement of technology and the willingness of each platform to release development permission, we were able to link Google, Line, and web services by using algorithmic mechanisms. The advantage of this integration is that we do not need to spend considerable time and money to develop a system but use the existing advantages and convenience of these platforms to achieve a similar experience. Moreover, as system developers, we are able to focus on the development and implementation of pedagogical theories rather than the basic operation and maintenance of the system.

To investigate the usability of the system and to help us improve the system, we will invite students and teachers as participants. This system is a prototype. Some message types follow a Line template, and thus, there are limitations, such as the number of buttons, length of the content, and appearance of the message. In addition, in the Google Sheet employed in this study, restrictions and drop-down lists cannot be implemented to prevent designers from constructing learning content with an incorrect format. Therefore, many functions need to be implemented and improved to make the system more accessible for designers. Moreover, because students’ data stored in Google Sheets cannot currently be read easily, the data must be organized; we expect to take the same Google Sheet format as the basis for developing another chatbot with which teachers can produce statistical analyses of students’ learning records.

The system is expected to be a tool that can help teachers understand how students make interpretations and inferences when reading a text. Especially for students who cannot obtain the correct understanding, the relationship between their explanations and text sentences can help teachers to counsel such students or help researchers analyze the factors causing misunderstanding. In the future, we expect to apply machine learning models to further distinguish and label students’ reading difficulties.

[ 1 ]     J. Fu, “Complexity of ICT in education: A critical literature review and its implications,”  International Journal of education and Development using ICT,  vol. 9, no. 1, pp. 112-125, 2013.

[ 2 ]    P. A. Ertmer, A. T. Ottenbreit-Leftwich, O. Sadik, E. Sendurur, and P. Sendurur, “Teacher beliefs and technology integration practices: A critical relationship,” Computers & education,  vol. 59, no. 2, pp. 423-435, 2012, DOI: 10.1016/j.compedu.2012.02.001

[ 3 ]     F. A. Inan and D. L. Lowther, “Factors affecting technology integration in K-12 classrooms: A path model,” Educational technology research and development,  vol. 58, no. 2, pp. 137-154, 2010, DOI: 1 0.1007/s11423-009-9132-y

[ 4 ]     W. Kintsch and C. Walter Kintsch,  Comprehension: A paradigm for cognition . Cambridge: Cambridge university press, 1998.

[ 5 ]     M. A. Britt, C. A. Perfetti, R. Sandak, and J.-F. Rouet, “Content integration and source separation in learning from multiple texts,” in  Narrative comprehension, causality, and coherence: Essays in honor of Tom Trabasso . Mahwah, NJ: Lawrence Erlbaum Associates, 1999,    pp. 209-233.

[ 6 ]     T. H. Nelson, “Complex information processing: a file structure for the complex, the changing and the indeterminate,” in  Proceedings of the 1965 20th national conference , 1965, pp. 84-100, DOI: 10.1145/800197.806036

[ 7 ]     T. A. Van Dijk and W. Kintsch,  Strategies of discourse comprehension , New York: Academic Press, 1983.

[ 8 ]     S. R. Goldman  et al. , “Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy,”  Educational Psychologist,  vol. 51, no. 2, pp. 219-246, 2016, DOI: 10.1080/00461520.2016.1168741

[ 9 ]     Ø. Anmarkrud, I. Bråten, and H. I. Strømsø, “Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents,”  Learning and Individual Differences,  vol. 30, pp. 64-76, 2014, DOI: 10.1016/j.lindif.2013.01.007

[ 10 ]   M. A. Britt and J.-F. Rouet, “Learning with multiple documents: Component skills and their acquisition,” in  Enhancing the quality of learning: Dispositions, instruction, and learning processes,  J. R. Kirby and M. J. Lawson, Eds. Cambridge: Cambridge University Press, 2012 ,  pp. 276-314.

[ 11 ]  T. J. Berners-Lee and R. Cailliau, “Worldwideweb: Proposal for a hypertext project,” CERN European Laboratory for Particle Physics, Nov. 1990.

[ 12 ]   M. Li  et al. , “Promoting reading comprehension and critical–analytic thinking: A comparison of three approaches with fourth and fifth graders,”  Contemporary Educational Psychology,  vol. 46, pp. 101-115, 2016, DOI: 10.1016/j.cedpsych.2016.05.002

[ 13 ]   W. J. Therrien and C. Hughes, “Comparison of Repeated Reading and Question Generation on Students’ Reading Fluency and Comprehension,”  Learning disabilities: A contemporary journal,  vol. 6, no. 1, pp. 1-16, 2008.

[ 14 ]   P. Afflerbach and B.-Y. Cho, “Identifying and describing constructively responsive comprehension strategies in new and traditional forms of reading,”  Handbook of research on reading comprehension . New York: Routledge, 2009,   pp. 69-90.

[ 15 ]   T. Andre, “Does answering higher-level questions while reading facilitate productive learning?,”  Review of Educational Research,  vol. 49, no. 2, pp. 280-318, 1979, DOI:  0.3102/00346543049002280

[ 16 ]   S. Degener and J. Berne, “Complex questions promote complex thinking,”  The Reading Teacher,  vol. 70, no. 5, pp. 595-599, 2017, DOI: 10.1002/trtr.1535

[ 17 ]   OECD. PISA 2018 assessment and analytical framework, Paris: OECD Publishing, 2019.

[ 18 ]   B. M. Taylor, P. D. Pearson, K. F. Clark, and S. Walpole, “Beating the Odds in Teaching All Children To Read,” Center for the Improvement of Early Reading Achievement, University of Michigan, Ann Arbor, Michigan, CIERA-R-2-006, 1999.

[ 18 ]   B. M. Taylor, P. D. Pearson, D. S. Peterson, and M. C. Rodriguez, “Reading growth in high-poverty classrooms: The influence of teacher practices that encourage cognitive engagement in literacy learning,”  The Elementary School Journal,  vol. 104, no. 1, pp. 3-28, 2003, DOI: 10.1086/499740

[ 19 ]   B. M. Taylor, P. D. Pearson, D. S. Peterson, and M. C. Rodriguez, “Reading growth in high-poverty classrooms: The influence of teacher practices that encourage cognitive engagement in literacy learning,”  The Elementary School Journal,  vol. 104, no. 1, pp. 3-28, 2003, DOI:  10.1086/499740

[ 20 ]   E. A. O’connor and M. C. Fish, “Differences in the Classroom Systems of Expert and Novice Teachers,” in  the meetings of the American Educational Research Association , 1998.

[ 21 ]   D. Linvill, “Student interest and engagement in the classroom: Relationships with student personality and developmental variables,”  Southern Communication Journal,  vol. 79, no. 3, pp. 201-214, 2014, DOI: 10.1080/1041794X.2014.884156

[ 22 ]   G. Davenport and B. Bradley, “The care and feeding of users,”  IEEE multimedia,  vol. 4, no. 1, pp. 8-11, 1997, DOI: 10.1109/93.580390

[ 23 ]   A. Rastogi, X. Zang, S. Sunkara, R. Gupta, and P. Khaitan, “Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset,” in  Proceedings of the AAAI Conference on Artificial Intelligence , 2020, vol. 34, no. 05, pp. 8689-8696, DOI: 10.1609/aaai.v34i05.6394

[ 24 ]   L. Fryer and R. Carpenter, “Bots as language learning tools,”  Language Learning & Technology,  vol. 10, no. 3, pp. 8-14, 2006, DOI: 10125/44068

[ 25 ]   L. K. Fryer, K. Nakao, and A. Thompson, “Chatbot learning partners: Connecting learning experiences, interest and competence,”  Computers in Human Behavior,  vol. 93, pp. 279-289, 2019, DOI: 10.1016/j.chb.2018.12.023

[ 26 ]   J. Yin, T.-T. Goh, B. Yang, and Y. Xiaobin, “Conversation technology with micro-learning: The impact of chatbot-based learning on students’ learning motivation and performance,”  Journal of Educational Computing Research,  vol. 59, no. 1, pp. 154-177, 2021, DOI: 10.1177/0735633120952067

[ 27 ]   P. Tracey, M. Saraee, and C. Hughes, “Applying NLP to build a cold reading chatbot,” in  2021 International Symposium on Electrical, Electronics and Information Engineering , 2021, pp. 77-80, DOI: 10.1145/3459104.3459119

[ 28 ]   W. Villegas-Ch, A. Arias-Navarrete, and X. Palacios-Pacheco, “Proposal of an Architecture for the Integration of a Chatbot with Artificial Intelligence in a Smart Campus for the Improvement of Learning,”  Sustainability,  vol. 12, no. 4, p. 1500, 2020, DOI: 10.3390/su12041500

This research project is grateful for the support of Taiwan Ministry of Science and Technology (MOST 110-2410-H-007-059-MY2.).

Wen-Hsiu Wu

Wen-Hsiu Wu

received her B.S. and M.S in Physics from National Tsing Hua University (NTHU, Taiwan, R.O.C.). She is currently continuing her research as a graduate student in the Institute of Learning Sciences and Technologies at National Tsing Hua University. Her research interests include digital learning environment analysis and design, specifically designing for cross-disciplinary learning and reading comprehension.

Guan-Ze Liao

Guan-Ze Liao

received his doctoral degree in Design programs at National Taiwan University of Science and Technology (NTUST). During his academic study at the university, he incorporated professional knowledge from various disciplines (e.g., multimedia, interaction design, visual and information design, arts and science, interactive media design, computer science, and information communication) into human interaction design, communication and media design research studies and applications. His cross-disciplinary research interests involve methods in information media research, interaction/interface design, multimedia game, and Computation on Geometric Patterns. Now he is a professor in the Institute of Learning Sciences and Technologies at National Tsing Hua University (NTHU, Taiwan, R.O.C.). His professional experience is focused in the fields of digital humanity, game-based learning, visual narrative, and information design, and the domain of design for learning.

Designing an Educational Chatbot: A Case Study of CikguAIBot

Nurul Amelina Nasharuddin , N. Sharef , E. Mansor + 6 more authors

Jun 15, 2021

Influential Citations

2021 Fifth International Conference on Information Retrieval and Knowledge Management (CAMP)

Key takeaway

Cikguaibot, a chatbot application for teaching ai in malay language, successfully achieved its objectives and is fostering successful learning in malaysia's education system..

This research aims to design a chatbot application to teach Artificial Intelligent (AI) in Malay language. CikguAIBot offers learners the possibility to learn and interact with an agent without the need for a teacher to be around. The development of CikguAIBot is based on the RAD model with the involvement of a number of experts and real users. The main focus of this paper is on the contents and flow design of the chatbot so that the objectives of the chatbot are achieved. Results from the expert review sessions were reported and a detailed evaluation strategy with the students is also included although the evaluation session is in the future plan. This research is expected to foster the usage of chatbot technology in supporting successful learning in Malaysia’s education system.

What We've Learned from Designing AI Chatbots for Education

Running a startup is known to be a long and arduous journey, but the Noodle Factory team has kept at it because we know that chatbots can help to change the world . As we’ve grown in knowledge, skills, and clientele, we’ve learned a lot to help us create better AI chatbots for our customers.

From trends to festive seasons to the usual FAQs , there are countless things to consider before building an AI chatbot for education. Through trial and error, conversations with industry experts, and experience with educational bodies, we’ve come to realise that there are a few key questions that need answering before you can embark on the chatbot designing process.

Keep reading to find out what they are!

1. What are your learning objectives?

First things first, you have to define the learning objectives you want to achieve with your chatbot. With any project, the most important thing to do is to clearly identify its purpose. In the case of an AI chatbot for education: What do you want your learners to take home?

2.Who is your target audience?

Phone and tablet brands have created numerous versions in order to cater to every possible audience group. There are high-tech options for the tech-savvy, easy interfaces for the older generation, and parental-controlled devices for young children. Similarly, AI chatbots have to be designed with a clear audience group in mind. Only then will you be able to ensure your chatbot speaks their language and lingo while providing the necessary information.

Is your chatbot going to entertain young children in primary school? Help teens in secondary school? Or maybe have informative conversations with tertiary students? Depending on your target audience, your chatbot needs will vary greatly.

For example, corporate learners require microlearning and just-in-time learning that happens outside their work and personal appointments, while higher education students require a transfer of knowledge with global scopes with an emphasis on evaluation and grading of student knowledge.

3.Will the chatbot be the primary form of learning?

You also have to decide on your classroom setting: Will it be a physical or online classroom? Will your chatbot be the only learning tool? Knowing this, you can determine what the chatbot need to achieve, and then you can find the chatbot that fits your need.

4.Do you want students to return to the chatbot?

Most conversational text and speech chatbot replies rely on NLP, so while it may be tempting to look into chatbots engineered with basic flow charts due to their low cost, NLP-engineered chatbots are significantly more human-like.

A chatbot that supports NLP will give the best results in understanding user intent, finding and remembering patterns, and replying with relevant content.

Final takeaway

Most importantly, if you want to create an AI chatbot that is truly stimulating and engaging to people, the chatbot should be able to do the same tasks that you or a human teacher can!

Similar posts

How to build feminist chatbot designs, 5 ways to prepare chatbots for the holidays: chinese new year edition, chatbots for learning: 4 reasons why your online classes are failing, 📧 subscribe to our newsletter.

Join our community of forward-thinkers and be the first to know about platform updates, exclusive promotions, upcoming events, and the latest news. Transform the way you teach with insights right to your inbox.

  • Roosevelt Library
  • Research Guides

Understanding and Using ChatGPT Effectively

  • Positive Case Studies Involving AI Chatbots
  • Ethical Considerations
  • Negative Case Studies Involving AI Chatbots
  • Writing Good Prompts
  • Citation of AI This link opens in a new window

These case studies show the positive impact AI chatbots have had on various industries.

  • IBM Watson for Oncology is an AI chatbot that assists oncologists in providing personalized treatment recommendations for cancer patients.
  • Case Study: In India, doctors at the Manipal Hospitals used IBM Watson for Oncology to analyze patient records and medical literature to recommend treatment plans. It resulted in improved treatment accuracy and patient outcomes, showcasing the potential of AI in healthcare.
  • Woebot is an AI chatbot designed to provide mental health support and cognitive-behavioral therapy (CBT) techniques to users.
  • Case Study: A study published in the Journal of Medical Internet Research (JMIR) found that using Woebot led to significant reductions in symptoms of depression and anxiety among college students. The chatbot's 24/7 availability and non-judgmental approach were noted as key benefits.
  • Duolingo, a language-learning app, introduced AI-powered chatbots to help learners practice conversational skills.
  • Case Study: Duolingo's chatbots have enhanced language learning by providing interactive and engaging conversations in various languages. Users reported increased confidence and fluency in their target languages.
  • Many businesses have integrated AI chatbots like ChatGPT into their customer support systems to provide quick and efficient assistance.
  • Case Study: Companies like Shopify and Adobe have successfully implemented AI chatbots to handle routine customer inquiries, resulting in reduced response times, improved customer satisfaction, and cost savings.
  • ScribeSense is an AI chatbot that assists students with disabilities by providing real-time transcriptions of classroom lectures.
  • Case Study: Universities and schools have adopted ScribeSense to make education more accessible. It has empowered students with hearing impairments or learning disabilities to participate fully in lectures and discussions.
  • Reuters, a global news agency, launched a chatbot that engages users with news-related trivia questions and provides news updates.
  • Case Study: The Reuters News Trivia Chatbot has made news consumption more interactive and engaging, especially among younger audiences. It encourages users to stay informed about current events.
  • << Previous: Ethical Considerations
  • Next: Negative Case Studies Involving AI Chatbots >>
  • Last Updated: Mar 24, 2024 1:30 PM
  • URL: https://libguides.roosevelt.edu/c.php?g=1349194

IMAGES

  1. Education Chatbot: Features & Use Cases

    designing an educational chatbot a case study of cikguaibot

  2. AI

    designing an educational chatbot a case study of cikguaibot

  3. The best AI chatbots for education

    designing an educational chatbot a case study of cikguaibot

  4. Designing Chatbots: A case study on Behance

    designing an educational chatbot a case study of cikguaibot

  5. Designing Chatbots: A case study on Behance

    designing an educational chatbot a case study of cikguaibot

  6. Designing Chatbots: A case study on Behance

    designing an educational chatbot a case study of cikguaibot

VIDEO

  1. UNIT

  2. Bay Area-developed AI chatbot tutor 'Khanmigo' aims to transform student learning

  3. ChatGPT in Education: Exploring the Impact of Generative AI on Teaching and Learning

  4. EduBot

  5. SchoolAI: Customized ChatBots for Your Class

  6. LESSONS LEARNED BUILDING A GENAI POWERED APP

COMMENTS

  1. Designing an Educational Chatbot: A Case Study of CikguAIBot

    This research aims to design a chatbot application to teach Artificial Intelligent (AI) in Malay language. CikguAIBot offers learners the possibility to learn and interact with an agent without the need for a teacher to be around. The development of CikguAIBot is based on the RAD model with the involvement of a number of experts and real users. The main focus of this paper is on the contents ...

  2. Designing an Educational Chatbot: A Case Study of CikguAIBot

    This study examined educational chatbots for Facebook Messenger to support learning. The independent web directory was screened to assess chatbots for this study resulting in the identification of ...

  3. Educational Design Principles of Using AI Chatbot That Supports Self

    Educational chatbots, also called conversational agents, hold immense potential in delivering personalized and interactive learning experiences to students [1,2].However, the advent of ChatGPT or generative AI poses a substantial challenge to the role of educators, as it gives rise to concerns that students may exploit generative AI tools to obtain academic recognition without actively ...

  4. Educational Design Principles of Using AI Chatbot

    We, therefore, propose an educational framework for integrating an AI educational chatbot based on learning science—Zimmerman's SRL framework along with JOL. To our best knowledge, the design of chatbots has focused greatly on the backend design [43], user interface [44], and improving learning [36,45,46].

  5. How to design an educational chatbot: Insights from experienced

    Case 1 served as the pilot study and focused on the effects of a chatbot conversation on the discussion; the critical thinking, satisfaction, and number of con-versations in experimental and ...

  6. Designing an Educational Chatbot: A Case Study of CikguAIBot

    Article "Designing an Educational Chatbot: A Case Study of CikguAIBot" Detailed information of the J-GLOBAL is an information service managed by the Japan Science and Technology Agency (hereinafter referred to as "JST"). It provides free access to secondary information on researchers, articles, patents, etc., in science and technology, medicine and pharmacy.

  7. Educational chatbots for project-based learning: investigating learning

    Educational chatbots (ECs) are chatbots designed for pedagogical purposes and are viewed as an Internet of Things (IoT) interface that could revolutionize teaching and learning. These chatbots are strategized to provide personalized learning through the concept of a virtual assistant that replicates humanized conversation. Nevertheless, in the education paradigm, ECs are still novel with ...

  8. Designing an Educational Chatbot: A Case Study of CikguAIBot

    Nurul Amelina Nasharuddin; Nurfadhlina Mohd Sharef; Evi Indriasari Mansor; Normalia Samian; Masrah Azrifah Azmi Murad; Mohd Khaizer Omar; Noreen Izza Arshad

  9. TRACE: A Conceptual Model to Guide the Design of Educational Chatbots

    Over the past decade, an increasing interest in integrating conversational agents into educational contexts has motivated the design, deployment, and evaluation of pedagogical conversational agents, also referred to as educational chatbots.Indeed, there is a wide variety of educational chatbot designs and architectures, with one review of the literature noting that "there exists as much ...

  10. Exploring the Pedagogical Use of AI-Powered Chatbots Educational

    In this research paper, we examine the current state of AI-powered chatbots in education and discuss the advantages and disadvantages of using chatbots in this context. A case study of an AI unit planner (Copilot) built on top of the GPT-3 application in K-12 settings will also be presented, examining how chatbots are used to assist teachers in ...

  11. Integrating chatbots in education: insights from the Chatbot-Human

    Recent advances in Artificial Intelligence (AI) have paved the way for the integration of text-based and voice-enabled chatbots as adaptive virtual tutors in education. Despite the increasing use of AI-powered chatbots in language learning, there is a lack of studies exploring the attitudes and perceptions of teachers and students towards these intelligent tutors. This study aims to compare ...

  12. Designing an Educational Chatbot: A Case Study of CikguAIBot

    This research aims to design a chatbot application to teach Artificial Intelligent (AI) in Malay language and foster the usage of chatbot technology in supporting successful learning in Malaysia's education system. This research aims to design a chatbot application to teach Artificial Intelligent (AI) in Malay language. CikguAIBot offers learners the possibility to learn and interact with an ...

  13. Cikguaibot: a Chatbot to Teach Artificial Intelligence in Malay

    The purpose of the CikguAIBot is to provide an alternative to learning materials and interaction modality with the instructor. The target user of the chatbot ranges from secondary school learners to lifelong learners. CikguAIBot is deployed as a Telegram application and executable through mobile apps and web access.

  14. PDF Exploring the Trend and Potential Distribution of Chatbot in Education

    and update the most current research on chatbots in education. These findings may give researchers and educators, as well as policymakers, with an overview of the research on chatbots in education. This systematic literature review is to investigate the current state of study in educational Chatbot/Intelligent agent in classrooms, focusing on ...

  15. Developing Effective Educational Chatbots with ChatGPT prompts

    We present a case study with a simple system that enables mixed-turn chatbot interactions and discuss the insights and preliminary guidelines obtained from initial tests. ... Educational chatbots come with a promise of interactive and personalized learning experiences, yet their development has been limited by the restricted free interaction ...

  16. ‪Nurul Amelina Nasharuddin‬

    ‪Senior Lecturer, Universiti Putra Malaysia‬ - ‪‪Cited by 250‬‬ - ‪Information Retrieval‬ - ‪Educational Technologies‬ - ‪Personalised Learning‬ - ‪Usability Studies‬ - ‪Multimedia Computing‬ ... Designing an Educational Chatbot: A Case Study of CikguAIBot. ... Game Design and Initial User Experiences. NA ...

  17. [2306.10645] Developing Effective Educational Chatbots with ChatGPT

    Educational chatbots come with a promise of interactive and personalized learning experiences, yet their development has been limited by the restricted free interaction capabilities of available platforms and the difficulty of encoding knowledge in a suitable format. Recent advances in language learning models with zero-shot learning capabilities, such as ChatGPT, suggest a new possibility for ...

  18. Design of a Chatbot Learning System: Case Study of a Reading Chatbot

    Therefore, technologies that can be easily used by teachers must be developed. In this study, an algorithm was developed that integrates Google Sheets with Line to offer teachers who are unfamiliar with programming a quick method for constructing a chatbot based on their teaching plan, question design, and the material prepared for a reading class.

  19. Designing an Educational Chatbot: A Case Study of CikguAIBot

    Key takeaway: 'CikguAIBot, a chatbot application for teaching AI in Malay language, successfully achieved its objectives and is fostering successful learning in Malaysia's education system.'

  20. PDF Student Experiences on the Use of Artificial Intelligence-Based

    Furthermore, Kumar (2021) highlighted the potential of educational chatbots as a pedagogical tool to revolutionize teaching and learning. In addition to language and science education, chatbots have been studied in the context of higher education, with Essel et al. (2022) focusing on the impact of a virtual ... Research Design In the study, a ...

  21. What We've Learned from Designing AI Chatbots for Education

    Similarly, AI chatbots have to be designed with a clear audience group in mind. Only then will you be able to ensure your chatbot speaks their language and lingo while providing the necessary information. Is your chatbot going to entertain young children in primary school? Help teens in secondary school?

  22. PDF An investigation of student's attitude towards use of chatbot

    Educational Research (ISSN: 2141-5161) Vol. 4(10) pp. 710-716, October, 2013 ... An investigation of student's attitude towards use of chatbot technology in Instruction: the case of Knowie in a selected high school Patrick Bii*1, Jackson Too 2, Reuben ... The study was a one-shot case study design in which a group was exposed to an event ...

  23. Positive Case Studies Involving AI Chatbots

    Education: ScribeSense. ScribeSense is an AI chatbot that assists students with disabilities by providing real-time transcriptions of classroom lectures. ... Case Study: The Reuters News Trivia Chatbot has made news consumption more interactive and engaging, especially among younger audiences. It encourages users to stay informed about current ...