2004
  1. Home
  2. A-Z Publications
  3. European Journal of Education Policy and Practice
  4. Volume 1, Issue 1
  5. Article
Volume 1, Issue 1
  • E-ISSN: 3051-1208
file format pdf download PDF

Abstract

Abstract

Since the 1990s, there have been heated debates about how evidence should be used to guide teaching practice and education policy, and how educational research can generate robust and trustworthy evidence. This paper reviews existing debates on evidence-based education and research on the impacts of AI in education and suggests a new conceptualisation of evidence aligned with an emerging learning-oriented model of science-for-policy, which we call S4P 3.0. Existing empirical evidence on AIED suggests some positive effects, but a closer look reveals methodological and conceptual problems and leads to the conclusion that existing evidence should not be used to guide policy or practice. AI is a new type of technology that interacts with human cognition, communication, and social knowledge infrastructures, and it requires rethinking what we mean by “learning outcomes” and policy and practice-relevant evidence. A common belief that AI-supported personalisation will “revolutionise” education is historically rooted in a methodological confusion that we call the Bloomian paradox in AIED, and based on a limited view on the social functions of education.

© Ilkka Tuomi
Loading

Article metrics loading...

/content/journals/10.5117/EJEP2025.1.001.TUOM
2025-08-01
2025-12-14
Loading full text...

Full text loading...

/deliver/fulltext/30511208/1/1/EJEP2025.1.001.TUOM.html?itemId=/content/journals/10.5117/EJEP2025.1.001.TUOM&mimeType=html&fmt=ahah

References

  1. Archer, M., Bhaskar, R., Collier, A., Lawson, T. & Norrie, A. (2013). Critical Realism: Essential Readings. Taylor and Francis. http://www.123library.org/book_details/?id=103049
     [Google Scholar]
  2. Arendt, H. (1961). Between Past and Future: Six Exercises in Political Thought. Penguin Books.
     [Google Scholar]
  3. Banerjee, A.V., & Duflo.E. (2009). The Experimental Approach to Development Economics. Annual Review of Economics, 1, 151–78. https://doi.org/10.1146/annurev.economics.050708.143235
     [Google Scholar]
  4. Barrett, C. B., & Carter, M.R. (2010). The Power and Pitfalls of Experiments in Development Economics: Some Non-Random Reflections. Applied Economic Perspectives and Policy, 32(4), 515–48. https://doi.org/10.1093/aepp/ppq023
     [Google Scholar]
  5. Bhaskar, R. (2013). A Realist Theory of Science. Routledge. https://doi.org/10.4324/9780203090732
     [Google Scholar]
  6. Biesta, G. (2007). Why “What Works” Won’t Work: Evidence-Based Practice and the Democratic Deficit in Educational Research. Educational Theory, 57(1), 1–22. https://doi.org/10.1111/j.1741-5446.2006.00241.x
     [Google Scholar]
  7. Biesta, G.2010. Good Education in an Age of Measurement: Ethics, Politics, Democracy. Routledge.
     [Google Scholar]
  8. Bloom, B. S. (1984). The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher, 13(6), 4–16. https://doi.org/10.3102/0013189X013006004
     [Google Scholar]
  9. Bond, M., HassanK., De Laat, M., Bergdahl, N., Negrea, V.Oxley, E., Pham, P., Chong, S.W. & Siemens, G. (2024). A Meta Systematic Review of Artificial Intelligence in Higher Education: A Call for Increased Ethics, Collaboration, and Rigour. International Journal of Educational Technology in Higher Education, 21(1), 4. https://doi.org/10.1186/s41239-023-00436-z
     [Google Scholar]
  10. Borenstein, M., Higgins, J.P.T., HedgesL.V. & Rothstein, H.R. (2017). Basics of Meta-Analysis: I2 Is Not an Absolute Measure of Heterogeneity. Research Synthesis Methods8(1), 5–18. https://doi.org/10.1002/jrsm.1230
     [Google Scholar]
  11. Cartwright, N. (2019). What Is Meant by “Rigour” in Evidence-Based Educational Policy and What’s so Good about It?Educational Research and Evaluation, 25(1–2), 63–80. https://doi.org/10.1080/13803611.2019.1617990
     [Google Scholar]
  12. Cheung, A. C. K. & Slavin, R.E. (2016). How Methodological Features Affect Effect Sizes in Education. Educational Researcher, 45(5), 283–92. https://doi.org/10.3102/0013189X16656615
     [Google Scholar]
  13. Coppe, T., Hoekstra, R., Helgetun, J.B.,Schwarzbach, N.R. & van der Velden, M. (2025). Evidence-Based Research in Education: The Questionable Epistemic Status of RCTs. Critical Studies in Education0(0), 1–17. https://doi.org/10.1080/17508487.2025.2501960
     [Google Scholar]
  14. Council of the European Union (2023). Strengthening the Role and Impact of Research and Innovation in the Policymaking Process in the Union, (16450/23). https://data.consilium.europa.eu/doc/document/ST-16450-2023-INIT/en/pdf
  15. Cowen, N. (2019). For Whom Does “What Works” Work? The Political Economy of Evidence-Based Education. Educational Research and Evaluation, 25(1–2), 81–98. https://doi.org/10.1080/13803611.2019.1617991
     [Google Scholar]
  16. Davies, P. (1999). What Is Evidence-Based Education?British Journal of Educational Studies, 47 (2), 108–21. https://doi.org/10.1111/1467-8527.00106
     [Google Scholar]
  17. Deaton, A., & Cartwright, N. (2018). Understanding and Misunderstanding Randomized Controlled Trials. Social Science & Medicine, 210(August), 2–21. https://doi.org/10.1016/j.socscimed.2017.12.005
     [Google Scholar]
  18. Ruiqi, D., Jiang, M., Yu, X., Lu,Y. & Liu, S. (2025). Does ChatGPT Enhance Student Learning? A Systematic Review and Meta-Analysis of Experimental Studies. Computers & Education, 227(April), 105224. https://doi.org/10.1016/j.compedu.2024.105224
     [Google Scholar]
  19. Derewianka, B., Harper, H., Parkin, B., Acevedo, C., Rose, D., Dare, F., Brisk, M.E. & Jones, P. (2024). What Counts as “Evidence” in Literacy Education?The Australian Journal of Language and Literacy, 47(3), 299–316. https://doi.org/10.1007/s44020-024-00071-9
     [Google Scholar]
  20. Dewey, J. (1916). Democracy and Education: An Introduction to the Philosophy of Education. The MacMillan Company.
     [Google Scholar]
  21. Dewey, J. (1958). Experience and Nature. Dover Publications, Inc.
     [Google Scholar]
  22. Doroudi, S. (2022). The Intertwined Histories of Artificial Intelligence and Education. International Journal of Artificial Intelligence in Education, October. https://doi.org/10.1007/s40593-022-00313-2
     [Google Scholar]
  23. EC (2022). Supporting and Connecting Policymaking in the Member States with Scientific Research (SWD 2022) 346 final. European Commission. https://knowledge4policy.ec.europa.eu/sites/default/files/SWD_2022_346_final.PDF.
  24. Engeström, Y. (1987). Learning by Expanding: An Activity Theoretical Approach to Developmental Work Research. Orienta Konsultit. https://lchc.ucsd.edu/mca/Paper/Engestrom/Learning-by-Expanding.pdf
     [Google Scholar]
  25. Escueta, M., Quan, V., Nickow, A.J. & Oreopoulos. P. (2017). Education Technology: An Evidence-Based Review (Working Paper 23744). National Bureau of Economic Research. https://doi.org/10.3386/w23744
     [Google Scholar]
  26. Fadel, C., Black, A., Taylor, R., Slesinski, J. & Dunn, K. (2024). Education for the Age of AI. Center for Curriculum Redesign. https://curriculumredesign.org/our-work/education-for-the-age-of-ai/
     [Google Scholar]
  27. Freire, P. (1972). Pedagogy of the Oppressed. Penguin.
     [Google Scholar]
  28. Funtowicz, S. O. & Ravetz, J.R. (1993). Science for the Post-Normal Age. Futures25(7), 739–55. https://doi.org/10.1016/0016-3287(93)90022-L
     [Google Scholar]
  29. Giroux, H. & McLaren, P. (2011). Teacher Education and the Politics of Engagement: The Case for Democratic Schooling. Harvard Educational Review, 56(3), 213–39. https://doi.org/10.17763/haer.56.3.trr1473235232320
     [Google Scholar]
  30. Hammersley, M. (2007). Educational Research and Evidence-Based Practice. SAGE. https://doi.org/10.4000/rechercheformation.673
     [Google Scholar]
  31. Hattie, J. (2008). Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. Routledge. https://doi.org/10.4324/9780203887332
     [Google Scholar]
  32. Heckman, J.J. (2020). Randomization and Social Policy Evaluation Revisited (NBER Technical Working Paper 107). National Bureau of Economic Research. https://www.nber.org/system/files/working_papers/t0107/t0107.pdf
     [Google Scholar]
  33. Holmes, W., Anastopoulou, S., Schaumburg, J. & Mavrikis. M. (2018). Technology-Enhanced Personalised Learning: Untangling the Evidence. Robert Bosch Stiftung. https://www.bosch-stiftung.de/sites/default/files/publications/pdf/2018-08/Study_Technology-enhanced%20Personalised%20Learning.pdf
     [Google Scholar]
  34. Inayatullah, S. (1998). Causal Layered Analysis: Poststructuralism as Method. Futures30(8), 815–29. https://doi.org/10.1016/S0016-3287(98)00086-X
     [Google Scholar]
  35. ISTE (2023). Teacher Ready: EdTech Product Evaluation Guide. International Society for Technology in Education. https://cdn.iste.org/www-root/2023-06/ISTE_Edtech_Product_Evaluation_Guide-2023_06.pdf
  36. Joyce, K. E. & Cartwright, N. (2018). Meeting Our Standards for Educational Justice: Doing Our Best with the Evidence. Theory and Research in Education, 16(1), 3–22. https://doi.org/10.1177/1477878518756565
     [Google Scholar]
  37. King, M., Rothberg, S., Dawson, R. & Batmaz, F. (2016). Bridging the Edtech Evidence Gap: A Realist Evaluation Framework Refined for Complex Technology Initiatives. Journal of Systems and Information Technology, 18(1), 18–40. https://doi.org/10.1108/JSIT-06-2015-0059
     [Google Scholar]
  38. Kloeg, J. (2022). Where Is Education? Arendt’s Educational Philosophy in between Private and Public. Journal of Philosophy of Education, 56(2), 196–209. https://doi.org/10.1111/1467-9752.12662
     [Google Scholar]
  39. Kluger, A.N., & DeNisi, A. (1996). The Effects of Feedback Interventions on Performance: A Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory. Psychological Bulletin, 119(2), 254–84. https://doi.org/10.1037/0033-2909.119.2.254
     [Google Scholar]
  40. Kosmyna, N., Hauptmann, E., Yuan, Y.T., Situ, J., Liao, X-H, Beresnitzky, A.V., Braunstein, I. & Maes, P. (2025). Your Brain on ChatGPT: Accumulation of Cognitive Debt When Using an AI Assistant for Essay Writing Task. arXiv. https://doi.org/10.48550/arXiv.2506.08872
     [Google Scholar]
  41. Kucirkova, N. & Dockterman. D. (2024). Towards a Holistic Understanding of Evidence: A Working Paper. UiS Scholarly Publishing Serviceshttps://static1.squarespace.com/static/62fc80ec4c86a26330d18835/t/66d1aee71febb01086d52da0/1725017833839/Towards+a+holistic+understanding+of+evidence+A+working+paper++%283%29.pdf
     [Google Scholar]
  42. Kucirkova, N., Schewe, O., Campbell, J., Cermakova, A.L. & Pitchford, N. (2025). Developing Evidence Indicators for Evaluating K12 EdTech: Towards a Consensus on Educational Impact. Humanities and Social Sciences Communications, 12(1), 947. https://doi.org/10.1057/s41599-025-05330-9
     [Google Scholar]
  43. Kulik, J. A. & Fletcher. J.D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review’. Review of Educational Research, 86(1), 42–78. https://doi.org/10.3102/0034654315581420
     [Google Scholar]
  44. Lemons, C. J., Fuchs, D., Gilbert, J.K, & Fuchs, L.S. (2014). Evidence-Based Practices in a Changing World: Reconsidering the Counterfactual in Education Research. Educational Researcher, 43(5), 242–52. https://doi.org/10.3102/0013189X14539189
     [Google Scholar]
  45. Ma, W., Adesope, O.O., Nesbit, J.C. & Liu, Q. (2014). Intelligent Tutoring Systems and Learning Outcomes: A Meta-Analysis. Journal of Educational Psychology, 106(4), 901–18. https://doi.org/10.1037/a0037123
     [Google Scholar]
  46. Mollick, E. R. & Mollick. L. (2023). Using AI to Implement Effective Teaching Strategies in Classrooms: Five Strategies, Including Prompts. SSRN Scholarly Paper. Social Science Research Network. https://doi.org/10.2139/ssrn.4391243
     [Google Scholar]
  47. Morrison, J. R, Borokhovski, E., Bernard, R.M. & Slavin, R.E. (2024). A Quantitative Synthesis of Outcomes of Educational Technology Approaches in K-12 Mathematics. Center for Research and Reform in Education, The John Hopkins University. https://bestevidence.org/wp-content/uploads/2024/09/edtechmathmorrisonetal2024-1.pdf
     [Google Scholar]
  48. Parra, J.,D. & Edwards, D.B. Jr. (2024). Challenging the Gold Standard Consensus: Randomised Controlled Trials (RCTs) and Their Pitfalls in Evidence-Based Education’. Critical Studies in Education, 65(5), 513–30. https://doi.org/10.1080/17508487.2024.2314118
     [Google Scholar]
  49. Pawson, R. & Tilley, N. (1997). Realistic Evaluation. SAGE. https://uk.sagepub.com/en-gb/eur/realistic-evaluation/book205276
     [Google Scholar]
  50. Piaget, J. & Inhelder, B. (1962). The Psychology Of The Child. Basic Books. https://www.hachettebookgroup.com/titles/jean-piaget/the-psychology-of-the-child/9780465095001/?lens=basic-books
     [Google Scholar]
  51. Rosen, R. (1978). Fundamentals of Measurement and Representation of Natural Systems. North-Holland.
     [Google Scholar]
  52. Salomon, G., ed. (1993). Distributed Cognitions: Psychological and Educational Considerations. Cambridge University Press.
     [Google Scholar]
  53. Simpson, A. (2019). The Evidential Basis of “Evidence-Based Education”: An Introduction to the Special Issue. Educational Research and Evaluation, 25(1–2), 1–6. https://doi.org/10.1080/13803611.2019.1617979
     [Google Scholar]
  54. Slavin, R. E. (1987). Mastery Learning Reconsidered. Review of Educational Research, 57(2), 175–213. https://doi.org/10.3102/00346543057002175
     [Google Scholar]
  55. Slavin, R. E. (2002). Evidence-Based Education Policies: Transforming Educational Practice and Research. Educational Researcher, 31(7), 15–21. https://doi.org/10.3102/0013189X031007015
     [Google Scholar]
  56. Slavin, R. E. (2008). Perspectives on Evidence-Based Research in Education—What Works? Issues in Synthesizing Educational Program Evaluations. Educational Researcher, 37(1), 5–14. https://doi.org/10.3102/0013189X08314117
     [Google Scholar]
  57. Slavin, R. E. & Neitzel, A. (2020). In Meta-Analyses, Weak Inclusion Standards Lead to Misleading Conclusions. Here’s Proof. Robert Slavin’s Blog (blog). 19November2020. https://robertslavinsblog.wordpress.com/2020/11/19/in-meta-analyses-weak-inclusion-standards-lead-to-misleading-conclusions-heres-proof/
     [Google Scholar]
  58. Steenbergen-Hu, S., & Cooper, H. (2013). A Meta-Analysis of the Effectiveness of Intelligent Tutoring Systems on K–12 Students’ Mathematical Learning. Journal of Educational Psychology, 105(4), 970–87. https://doi.org/10.1037/a0032447
     [Google Scholar]
  59. Steenbergen-Hu, S., & Cooper, H. (2014). A Meta-Analysis of the Effectiveness of Intelligent Tutoring Systems on College Students’ Academic Learning. Journal of Educational Psychology, 106(2), 331–47. https://doi.org/10.1037/a0034752
     [Google Scholar]
  60. Šucha, V. & Sienkiewicz, M. (2020). Science for Policy Handbook. Elsevier. https://shop.elsevier.com/books/science-for-policy-handbook/sucha/978-0-12-822596-7
     [Google Scholar]
  61. Thurow, L. C. (1975). Generating Inequality: Mechanisms of Distribution in the U.S. Economy. Basic Books.
     [Google Scholar]
  62. Tikly, L. (2015). What Works, for Whom, and in What Circumstances? Towards a Critical Realist Understanding of Learning in International and Comparative Education. International Journal of Educational Development 4 (January), 237–49. https://doi.org/10.1016/j.ijedudev.2014.11.008
     [Google Scholar]
  63. Toulmin, S. E. (2003). The Uses of Argument. 2nd ed. Cambridge University Press. https://doi.org/10.1017/CBO9780511840005
     [Google Scholar]
  64. Tuomi, I. (1999). Corporate Knowledge: Theory and Practice of Intelligent Organizations. Metaxis.
     [Google Scholar]
  65. Tuomi, I. (2002). Networks of Innovation: Change and Meaning in the Age of the Internet. Oxford University Press.
     [Google Scholar]
  66. Tuomi, I. (2015). Epistemic Literacy or a Clash of Clans? A Capability-Based View on the Future of Learning and Education. European Journal of Education, 50(1), 21–24. https://doi.org/10.1111/ejed.12101
     [Google Scholar]
  67. Tuomi, I. (2024a). Beyond Mastery: Toward a Broader Understanding of AI in Education. International Journal of Artificial Intelligence in Education, 34 (1), 20–30. https://doi.org/10.1007/s40593-023-00343-4
     [Google Scholar]
  68. Tuomi, I. (2024b). Fostering Knowledge-Sharing within and among S4P Actors: Mutual Learning Exercise on Bridging the Gap between Science and Policy. Publications Office of the European Union. https://data.europa.eu/doi/10.2777/4531314
     [Google Scholar]
  69. Tuomi, I. (2025a). Adding Intelligence to AI&ED Policy and Practice. In AI and the Future of Learning (Tentative Title). UNESCO.
     [Google Scholar]
  70. Tuomi, I. (2025b). Mutual Learning Exercise on Bridging the Gap between Science and Policy: Final Report. European Commission. http:/doi.org/10.2777/5299851
     [Google Scholar]
  71. Tuomi, I., Cachia, R. & Villar-Onrubia. D. (2023). On the Futures of Technology in Education: Emerging Trends and Policy Challenges. JRC Science for Policy Reports. European Commission. https://data.europa.eu/doi/10.2760/079734
     [Google Scholar]
  72. UNESCO. (2021). Reimagining Our Futures Together: A New Social Contract for Education. UNESCO. https://doi.org/10.54675/ASRB4722
     [Google Scholar]
  73. Vanbecelaere, S., Adam, T., Sieber, C., Clark-Wilson, A., Boody Adorno, K. & Haßler, B. (2023). Towards Systemic EdTech Testbeds: A Global Perspective. Global Edtech Testbed Network. https://doi.org/10.53832/opendeved.0285
     [Google Scholar]
  74. VanLehn, K. (2011). The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems. Educational Psychologist, 46(4), 197–221. https://doi.org/10.1080/00461520
     [Google Scholar]
  75. von Schomberg, R. (2025). Trust as a Governance Challenge for Science-for-Policy Ecosystems: Mutual Learning Exercise on Bridging the Gap between Science and Policy : Fourth Thematic Report. Publications Office of the European Union. https://data.europa.eu/doi/10.2777/6569914
     [Google Scholar]
  76. Vygotsky, L. (1986). Thought and Language. The MIT Press.
     [Google Scholar]
  77. Wang, J. & Fan, W. (2025). The Effect of ChatGPT on Students’ Learning Performance, Learning Perception, and Higher-Order Thinking: Insights from a Meta-Analysis. Humanities and Social Sciences Communications, 12(1), 1–21. https://doi.org/10.1057/s41599-025-04787-y
     [Google Scholar]
  78. WEF. (2024). Shaping the Future of Learning: The Role of AI in Education 4.0. World Economic Forum. https://www.weforum.org/publications/shaping-the-future-of-learning-the-role-of-ai-in-education-4-0/
  79. Weidlich, J., D.Gašević, H.Drachsler, and P.Kirschner. 2025. ‘ChatGPT in Education: An Effect in Search of a Cause’. Journal of Computer Assisted Learning41 (5): e70105. https://doi.org/10.1111/jcal.70105
     [Google Scholar]
  80. What Works Clearinghouse (2022). What Works Clearinghouse Procedures and Standards Handbook, Version 5.0. U.S. Department of Education, Institute of Education Sciences. https://ies.ed.gov/ncee/wwc/Handbooks
  81. Wiliam, D. (2019). Some Reflections on the Role of Evidence in Improving Education. Educational Research and Evaluation, 25(1–2), 127–39. https://doi.org/10.1080/13803611.2019.1617993
     [Google Scholar]
  82. Woolf, B.P. (2009). Building Intelligent Interactive Tutors: Student-Centered Strategies for Revolutionizing e-Learning. Morgan Kaufmann.
     [Google Scholar]
  83. Zawacki-Richter, O., Marín, V.I., Bond, M. & Gouverneur, F. (2019). Systematic Review of Research on Artificial Intelligence Applications in Higher Education – Where Are the Educators?’; International Journal of Educational Technology in Higher Education, 16(1), 39. https://doi.org/10.1186/s41239-019-0171-0
     [Google Scholar]
/content/journals/10.5117/EJEP2025.1.001.TUOM
Loading
Keyword(s): AIED impact; credibility of evidence; epistemology of research-informed policy; impact assessment; objectives of education; ontology of education; policy learning
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error