Abstract
Intelligent Tutoring Systems (ITSs) have a great potential to effectively transform teaching and learning. As more efforts have been put on designing and developing ITSs and integrating them within learning and instruction, mixed types of results about the effectiveness of ITS have been reported. Therefore, it is necessary to investigate how ITSs work in real and natural educational contexts and the associated challenges of ITS application and evaluation. Through a systematic literature review method, this study analyzed 40 qualified studies that applied social experiment methods to examine the effectiveness of ITS during 2011–2022. The obtained results highlighted a complicated landscape regarding the effectiveness of ITS in real educational contexts. Specifically, there was an “intelligent” regional gap regarding the distribution of countries where ITS studies using social experiment methods were conducted. Compared to learning performance, relatively less attention was paid to investigating the impact of ITS on non-cognitive factors, process-oriented factors, and social outcomes, calling for more research in this regard. Considering the complexities and challenges existing in real educational fields, there was a lack of scientific rigor in terms of experimental design and data analysis in some of the studies. Based on these findings, suggestions for future study and implications were proposed.




Similar content being viewed by others
Data availability
The datasets generated during the current study are available from the corresponding author on reasonable request.
References
Ahmed, H., Wilson, A., Mead, N., Noble, H., Richardson, U., Wolpert, M. A., & Goswami, U. (2020). An evaluation of the efficacy of GraphoGame rime for promoting English phonics knowledge in poor readers. Frontiers in Education, 5, 132. https://doi.org/10.3389/feduc.2020.00132
Alabdulhadi, A., & Faisal, M. (2021). Systematic literature review of STEM self-study related ITSs. Education and Information Technologies, 26(2), 1549–1588. https://doi.org/10.1007/s10639-020-10315-z
Alshammari, M. T., & Qtaish, A. (2019). Effective adaptive e-learning systems according to learning style and knowledge level. Journal of Information Technology Education, 18, 529–547. Retrieved December 29th, 2022 from http://www.jite.org/documents/Vol18/JITEv18ResearchP529-547Alshammari5698.pdf
Atun, H. (2020). Intelligent tutoring systems (its) to improve reading comprehension: a systematic review. Journal of Teacher Education and Lifelong Learning, 2(2), 77–89. Retrieved December 29th, 2022 from https://dergipark.org.tr/en/pub/tell/issue/58491/757329
Baker, D. L., Ma, H., Polanco, P., Conry, J. M., Kamata, A., Al Otaiba, S., Ward, W., & Cole, R. (2020). Development and promise of a vocabulary intelligent tutoring system for second-grade Latinx English learners. Journal of Research on Technology in Education, 53(2), 223–247. https://doi.org/10.1080/15391523.2020.1762519
Bartelet, D., Ghysels, J., Groot, W., Haelermans, C., & Maassen van den Brink, H. (2016). The differential effect of basic mathematics skills homework via a web-based intelligent tutoring system across achievement subgroups and mathematics domains: A randomized field experiment. Journal of Educational Psychology, 108(1), 1–20. https://doi.org/10.1037/edu0000051
Bernacki, M. L., & Walkington, C. (2018). The role of situational interest in personalized learning. Journal of Educational Psychology, 110(6), 864–881. https://doi.org/10.1037/edu0000250
Burns, H.L., & Capps, C.G. (1988). Foundations of intelligent tutoring systems: An introduction. In M. C. Polson & J. J. Richardson (Eds.), Foundations of intelligent tutoring systems (pp. 1–19). Lawrence Erlbaum. https://www.taylorfrancis.com/chapters/mono/10.4324/9780203761557-6/foundations-intelligent-tutoring-systems-introduction-martha-polson-jeffrey-richardson
Butcher, K. R., & Aleven, V. (2013). Using student interactions to foster rule–diagram mapping during problem solving in an intelligent tutoring system. Journal of Educational Psychology, 105(4), 988–1009. https://doi.org/10.1037/a0031756
Capone, R., De Falco, M., & Lepore, M. (2022). The Impact of Covid-19 Pandemic on Undergraduate Students: the Role of an Adaptive Situation-Aware Learning System. In 2022 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA) (pp. 154-161). IEEE.
Carter, E. E. (2014). An intelligent debugging tutor for novice computer science students. [Doctoral dissertation, Lehigh University]. ProQuest Dissertations & Theses Global. Retrieved December 29th, 2022 from https://www.proquest.com/dissertations-theses/intelligent-debugging-tutor-novice-computer/docview/1540757322/se-2
Casas, I., Goodman, P.S., Pelaez, E. (2011). On the design and use of a cognitive tutoring system in the math classroom [Paper presentation]. 2011 IEEE International Conference on Technology for Education, Chennai, Tamil Nadu India. Retrieved December 29th, 2022 from https://ieeexplore.ieee.org/document/6004354
Chang, Y. H., Chen, Y. Y., Chen, N. S., Lu, Y. T., & Fang, R. J. (2016). Yet another adaptive learning management system based on Felder and Silverman’s learning styles and Mashup. Eurasia Journal of Mathematics, Science and Technology Education, 12(5), 1273–1285. http://www.ejmste.com/ms.aspx?kimlik=10.12973/eurasia.2016.1512a
Churi, P. P., Joshi, S., Elhoseny, M., & Omrane, A. (Eds.). (2022). Artificial intelligence in higher education: A practical approach (1st ed.). CRC Press. https://doi.org/10.1201/9781003184157
Colby, B. R. (2017). A comparative literature review of intelligent tutoring systems from 1990–2015. [Master's thesis, Brigham Young University]. Scholars Archive. Retrieved December 29th, 2022 from https://scholarsarchive.byu.edu/etd/7239/
Corbett, A. T., Koedinger, K., & Hadley, W. S. (2001). Cognitive tutors: From the research classroom to all classrooms. In P. S. Goodman (Ed.), Technology enhanced learning: Opportunities for change (pp. 235–263). Lawrence Erlbaum Associates Publishers.
Craig, S. D., Hu, X., Graesser, A. C., Bargagliotti, A. E., Sterbinsky, A., Cheney, K. R., & Okwumabua, T. (2013). The impact of a technology-based mathematics after-school program using ALEKS on student’s knowledge and behaviors. Computers & Education, 68, 495–504. https://doi.org/10.1016/j.compedu.2013.06.010
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018, January 30- February 2). Intelligent tutoring systems for programming education: a systematic review [Paper presentation]. 20th Australasian Computing Education Conference, Brisbane, Queensland, Australia.
Cuéllar-Rojas, O. A., Hincapié, M., Contero, M., & Güemes-Castorena, D. (2021). Intelligent tutoring system: A bibliometric analysis and systematic literature review. Research Square. Advance online publication. https://doi.org/10.21203/rs.3.rs-673038/v1
Cung, B., Xu, D., Eichhorn, S., & Warschauer, M. (2019). Getting academically underprepared students ready through college developmental education: Does the course delivery format matter? American Journal of Distance Education, 33(3), 178–194. https://doi.org/10.1080/08923647.2019.1582404
del Olmo-Muñoz, J., González-Calero, J. A., Diago, P. D., Arnau, D., & Arevalillo-Herráez, M. (2022). Intelligent tutoring systems for word problem solving in COVID-19 days: could they have been (part of) the solution? ZDM–Mathematics Education, 1–14.
Eryilmaz, M., & Adabashi, A. (2020). Development of an intelligent tutoring system using bayesian networks and fuzzy logic for a higher student academic performance. Applied Sciences, 10(19), 6638. https://doi.org/10.3390/app10196638
Fang, N., & Guo, Y. (2013). A web-based interactive intelligent tutoring system for undergraduate engineering dynamics [Paper presentation]. 2013 IEEE Frontiers in Education Conference, Oklahoma City, USA.
Feng, M., Roschelle, J., Heffernan, N., Fairman, J., & Murphy, R. (2014). Implementation of an intelligent tutoring system for online homework support in an efficacy trial [Paper presentation]. 12th International Conference on Intelligent Tutoring Systems, Verlag, Berlin, Heidelberg.
Feng, S., Magana, A. J., & Kao, D. (2021). A systematic review of literature on the effectiveness of intelligent tutoring systems in STEM [Paper presentation]. 2021 IEEE Frontiers in Education Conference (FIE), Lincoln, NE, USA.
Fienberg S. E., Singer B., Tanur J.M. (1985). Large-Scale Social Experimentation in the United States. In A. C. Atkinson & S. E. Fienberg (Eds), A Celebration of Statistics (pp. 287–326). Springer, New York. https://doi.org/10.1007/978-1-4613-8560-8_12
Forget. (2019). Experiments in Society: Framing social experiments at the boundary between social work and sociology [Paper presentation]. Centre for the History of Political Economy (CHOPE), Durham, North Carolina, USA.
Goel, A. K., & Joyner, D. A. (2017). Using AI to teach AI: Lessons from an online AI class. AI Magazine, 38(2), 48–59. https://doi.org/10.1609/aimag.v38i2.2732
Greenberg, D., & Shroder, M. (2004). The digest of social experiments. Urban Institute Press.
Guha, M. (2008), International encyclopedia of the social sciences (2nd edition). Reference Reviews, 22(7), 17–19. https://doi.org/10.1108/09504120810905060
Harris, J. D., Quatman, C. E., Manring, M. M., Siston, R. A., & Flanigan, D. C. (2014). How to write a systematic review. The American Journal of Sports Medicine, 42(11), 2761–2768. https://doi.org/10.1177/0363546513497567
Hickey, D. T., Robinson, J., Fiorini, S., & Feng, Y. (2020). Internet-based alternatives for equitable preparation, access, and success in gateway courses. The Internet and Higher Education, 44, 100693. https://doi.org/10.1016/j.iheduc.2019.100693
Huang, X., Craig, S. D., Xie, J., Graesser, A. C., Okwumabua, T., Cheney, K. R., & Hu, X. (2013). The relationship between gender, ethnicity, and technology on the impact of mathematics achievement in an after-school program [Paper presentation]. Society for Research on Educational Effectiveness Spring 2013, Washington, D.C., USA.
Jiang, Y., Almeda, M., Kai, S., Baker, R. S., Ostrow, K., Inventado, P. S., & Scupelli, P. (2020). Single template vs. multiple templates: Examining the effects of problem format on performance [Paper presentation]. The 14th International Conference on the Learning Sciences, Nashville, Tennessee.
Kegel, C. A., & Bus, A. G. (2012). Online tutoring as a pivotal quality of web-based early literacy programs. Journal of Educational Psychology, 104(1), 182–192. https://doi.org/10.1037/A0025849
Keleş, A., Ocak, R., Keleş, A., & Gülcü, A. A. (2009). ZOSMAT: Web-based intelligent tutoring system for teaching–learning process. Expert Systems with Applications, 36(2), 1229–1239. https://doi.org/10.1016/j.eswa.2007.11.064
Kitchenham, B., & Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering (EBSE 2007–001). Retrieved December 29th, 2022 from https://www.researchgate.net/publication/302924724_Guidelines_for_performing_Systematic_Literature_Reviews_in_Software_Engineering
Klaveren, C., Vonk, S., & Cornelisz, I. (2017). The effect of adaptive versus static practicing on student learning-evidence from a randomized field experiment. Economics of Education Review, 58, 175–187. https://doi.org/10.1016/j.econedurev.2017.04.003
Koedinger, K. R., & Aleven, V. (2016). An interview reflection on “Intelligent tutoring goes to school in the big city”. International Journal of Artificial Intelligence in Education, 26(1), 13–24. https://link.springer.com/article/10.1007/s40593-015-0082-8
Lee, H. S., Anderson, J. R., Berman, S. R., Ferris-Glick, J., Joshi, A., Nixon, T., & Ritter, S. (2013). Exploring Optimal Conditions of Instructional Guidance in an Algebra Tutor [Paper presentation]. Society for Research on Educational Effectiveness Fall 2013, Washington, D.C., USA.
Li, K. C., & Wong, B. T. M. (2021). Features and trends of personalised learning: A review of journal publications from 2001 to 2018. Interactive Learning Environments, 29(2), 182–195. https://doi.org/10.1080/10494820.2020.1811735
Long, Y., & Aleven, V. (2017). Enhancing learning outcomes through self-regulated learning support with an open learner model. User Modeling and User-Adapted Interaction, 27(1), 55–88. https://doi.org/10.1007/s11257-016-9186-6
Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence unleashed: An argument for AI in education. Pearson Education.
Ma, W., Adesope, O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4), 901–918. https://doi.org/10.1037/a0037123
Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement: elaboration and explanation. BMJ: British Medical Journal, 349, g7647. https://doi.org/10.1136/bmj.g7647
Mousavi, A., Schmidt, M., Squires, V., & Wilson, K. (2021). Assessing the effectiveness of student advice recommender agent (SARA): The case of automated personalized feedback. International Journal of Artificial Intelligence in Education, 31(3), 603–621. https://doi.org/10.1007/s40593-020-00210-6
Mousavinasab, E., Zarifsanaiey, N., R. Niakan Kalhori, S., Rakhshan, M., Keikha, L., & Ghazi Saeedi, M. (2018). Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments, 29(1), 142–163.https://doi.org/10.1080/10494820.2018.1558257
Neagu, L. M., Rigaud, E., Travadel, S., Dascalu, M., & Rughinis, R. V. (2020, June 8–12). Intelligent tutoring systems for psychomotor training–a systematic literature review [Paper presentation]. International Conference on Intelligent Tutoring Systems 2020, Athens, Greece.
Nwana, H. S. (1990). Intelligent tutoring systems: An overview. Artificial Intelligence Review, 4, 251–277. https://doi.org/10.1007/BF00168958
Nye, B. D. (2015). Intelligent tutoring systems by and for the developing world: A review of trends and approaches for educational technology in a global context. International Journal of Artificial Intelligence in Education, 25, 177–203. https://doi.org/10.1007/s40593-014-0028-6
Nye, B. D., Pavlik, P. I., Windsor, A., Olney, A. M., Hajeer, M., & Hu, X. (2018). SKOPE-IT (Shareable Knowledge Objects as Portable Intelligent Tutors): Overlaying natural language tutoring on an adaptive learning system for mathematics. International Journal of STEM Education, 5(1), 1–20. https://doi.org/10.1186/s40594-018-0109-4
OECD.AI (2022), Visualisations powered by JSI using data from Preqin, Retrieved December 29th, 2022 from www.oecd.ai
Paladines, J., & Ramírez, J. (2020). A systematic literature review of intelligent tutoring systems with dialogue in natural language. IEEE Access, 8, 164246–164267. https://doi.org/10.1109/ACCESS.2020.3021383
Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of cognitive tutor algebra I at scale. Educational Evaluation and Policy Analysis, 36(2), 127–144.
Peck, L. R. (Ed.). (2017). Social experiments in practice: The what, why, when, where, and how of experimental design and analysis: New Directions for Evaluation, Number 152. John Wiley & Sons.
Riecken, H. W., & Boruch, R. F. (1974). Social Experimentation: A Method for Planning and Evaluating Social Intervention. Academic Press.
Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 21(2), 267–280. https://doi.org/10.1016/j.learninstruc.2010.07.004
Rolston, H. (2016). On the “why” of social experiments: Some lessons on overcoming barriers from 45 Years of social experiments. New Directions for Evaluation, 2016(152), 19–31. https://doi.org/10.1002/ev.20214
Slavin, R. E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational researcher, 15(9), 5–11. Retrieved December 29th, 2022 from https://www.jstor.org/stable/1174711
Social Experiment. (2008). In W. A. Darity, Jr. (Ed.), International Encyclopedia of the Social Sciences (2nd ed., Vol. 7, pp. 590–592). Macmillan Reference USA. Retrieved December 29th, 2022 from https://link.gale.com/apps/doc/CX3045302492/WHIC?u=cnbnu&sid=bookmark-WHIC&xid=8e9ed662
Soofi, A. A., & Ahmed, M. U. (2019). A systematic review of domains, techniques, delivery modes and validation methods for intelligent tutoring systems. International Journal of Advanced Computer Science and Applications, 10(3), 99–107. https://doi.org/10.14569/IJACSA.2019.0100312
Spichtig, A. N., Gehsmann, K. M., Pascoe, J. P., & Ferrara, J. D. (2019). The impact of adaptive, web-based, scaffolded silent reading instruction on the reading achievement of students in grades 4 and 5. The Elementary School Journal, 119(3), 443–467. https://doi.org/10.1086/701705
Tacoma, S., Drijvers, P., & Jeuring, J. (2020). Combined inner and outer loop feedback in an intelligent tutoring system for statistics in higher education. Journal of Computer Assisted Learning, 37(2), 319–332. https://doi.org/10.1111/jcal.12491
Treceño-Fernández, D., Calabia-Del-Campo, J., Bote-Lorenzo, M. L., Gómez-Sánchez, E., Luis-García, R., & Alberola-López, C. (2020). Integration of an intelligent tutoring system in a magnetic resonance simulator for education: Technical feasibility and user experience. Computer Methods and Programs in Biomedicine, 195, 105634. https://doi.org/10.1016/j.cmpb.2020.105634
Troussas, C., Krouska, A., & Sgouropoulou, C. (2021). A novel teaching strategy through adaptive learning activities for computer programming. IEEE Transactions on Education, 64(2), 103–109. https://doi.org/10.1109/TE.2020.3012744
U.S. Department of Education. (2010). Evaluation evidence-based practices in online learning meta-analysis and review of online learning studies. Office of Planning, Evaluation, and Policy Development.
Ward, W., Cole, R., Bolaños, D., Buchenroth-Martin, C., Svirsky, E., & Weston, T. (2013). My science tutor: A conversational multimedia virtual tutor. Journal of Educational Psychology, 105(4), 1115. https://doi.org/10.1037/a0031589
Ward, W., Cole, R., Bolanos, D., Buchenroth-Martin, C., Svirsky, E., Vuuren, S. V., ..., & Becker, L. (2011). My science tutor: A conversational multimedia virtual tutor for elementary school science. ACM Transactions on Speech and Language Processing (TSLP), 7(4), 1–29.https://doi.org/10.1145/1998384.1998392
Watkins, P. C., Caporal, J., Merville, C., Kouider, S., & Dehaene, S. (2020). Accelerating reading acquisition and boosting comprehension with a cognitive science-based tablet training. Journal of Computers in Education, 7(2), 183–212. https://doi.org/10.1007/s40692-019-00152-6
Wetzel, J., VanLehn, K., Butler, D., Chaudhari, P., Desai, A., Feng, J., ..., & van de Sande, B. (2017). The design and development of the dragoon intelligent tutoring system for model construction: lessons learned. Interactive Learning Environments, 25(3), 361–381https://doi.org/10.1080/10494820.2015.1131167
Wijekumar, K. K., Meyer, B. J., & Lei, P. (2012). Large-scale randomized controlled trial with 4th graders using intelligent tutoring of the structure strategy to improve nonfiction reading comprehension. Educational Technology Research and Development, 60(6), 987–1013. https://www.jstor.org/stable/23356890
Wijekumar, K. K., Meyer, B. J., & Lei, P. (2013). High-fidelity implementation of web-based intelligent tutoring system improves fourth and fifth graders content area reading comprehension. Computers & Education, 68, 366–379. https://doi.org/10.1016/j.compedu.2013.05.021
Wijekumar, K., Meyer, B. J., Lei, P. W., Lin, Y. C., Johnson, L. A., Spielvogel, J. A., Shurmatz, K. M., Ray, M., & Cook, M. (2014). Multisite randomized controlled trial examining intelligent tutoring of structure strategy for fifth-grade readers. Journal of Research on Educational Effectiveness, 7(4), 331–357. https://doi.org/10.1080/19345747.2013.853333
Wijekumar, K., Meyer, B. J., Lei, P., Beerwinkle, A. L., & Joshi, M. (2020). Supplementing teacher knowledge using web-based intelligent tutoring system for the text structure strategy to improve content area reading comprehension with fourth-and fifth-grade struggling readers. Dyslexia, 26(2), 120–136. https://doi.org/10.1002/dys.1634
Yao, C. B. (2017). Constructing a user-friendly and smart ubiquitous personalized learning environment by using a context-aware mechanism. IEEE Transactions on Learning Technologies, 10(1), 104–114. https://doi.org/10.1109/TLT.2015.2487977
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education–where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 1–27. https://doi.org/10.1186/s41239-019-0171-0
Zhang, B., & Jia, J. (2017). Evaluating an intelligent tutoring system for personalized math teaching [Paper presentation]. 2017 international symposium on educational technology, Hong Kong, China.
Acknowledgements
The current study isfunded by Guangdong University Online Open Course Committee (广东省本科高校在线开放课程指导委员会, Grant Number: 2022ZXKC397) and China Institue of Education and Social Development (中国教育与社会发展研究院, Grant Number: Gb2021013).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, H., Tlili, A., Huang, R. et al. Examining the applications of intelligent tutoring systems in real educational contexts: A systematic literature review from the social experiment perspective. Educ Inf Technol 28, 9113–9148 (2023). https://doi.org/10.1007/s10639-022-11555-x
Received:
Accepted:
Published:
Version of record:
Issue date:
DOI: https://doi.org/10.1007/s10639-022-11555-x


