Do AI Writing Tools Improve L2 Writing Performance? A Meta-Analysis of Experimental Evidence, Moderators, and Methodological Quality

Jinming Du

doi:10.61414/awr9xk46

Abstract

Artificial intelligence (AI) writing tools—including large language model (LLM)-based systems, automated writing evaluation (AWE) platforms, and AI-powered writing assistants—have become an increasingly prominent feature of second language (L2) writing pedagogy. Yet whether these tools reliably improve L2 writing performance, and under what instructional conditions, remains an open question. This study reports a systematic meta-analysis of experimental and quasi-experimental evidence on the effects of AI writing tools on L2 writing performance. A multi-database search of Scopus, Web of Science, ERIC, and ProQuest Dissertations & Theses, supplemented by forward citation tracing and targeted hand-searching, identified 34 eligible focal effect sizes from studies published between 2021 and 2026. Effect sizes were expressed as Hedges’ g. A verified subset of 22 effects with directly extractable statistics was estimated using restricted maximum-likelihood (REML) with Knapp–Hartung adjustment; an expanded model (k = 34) incorporating conversion-based effects served as a robustness analysis. The verified model produced a pooled effect of g = 0.714, 95% CI [0.592, 0.836], with low observed heterogeneity (I² = 24.3%). Exploratory moderator analyses indicated that multidimensional feedback designs, sustained interventions, particularly those lasting eight weeks or longer, were associated with larger effects, while single-session and lower-order-only feedback showed smaller gains. Methodological quality was generally adequate but uneven: many studies lacked delayed transfer measures and transparent effect-size reporting. These findings position AI writing tools as pedagogically contingent rather than inherently effective—their educational value depends critically on how feedback literacy, teacher mediation, and instructional design shape opportunities for noticing, revision, and transferable writing development.

References

1.
Ajabshir, Z. F., & Ebadi, S. (2023). The effects of automatic writing evaluation and teacher-focused feedback on CALF measures and overall quality of L2 writing across different genres. Asian-Pacific Journal of Second and Foreign Language Education, 8(1), 26. https://doi.org/10.1186/s40862-023-00201-9.
2.
Al-Raimi, M., Mudhsh, B. A., Al-Yafaei, Y., & Al-Maashani, S. (2024). Utilizing artificial intelligence tools for improving writing skills: Exploring Omani EFL learners’ perspectives. Forum for Linguistic Studies, 6(2), 1177. https://doi.org/10.59400/fls.v6i2.1177.
3.
Aladini, A., Ismail, S. M., Khasawneh, M. A. S., & Shakibaei, G. (2025). Self-directed writing development across computer/AI-based tasks: Unraveling the traces on L2 writing outcomes, growth mindfulness, and grammatical knowledge. Computers in Human Behavior Reports, 17, 100566. https://doi.org/10.1016/j.chbr.2024.100566.
4.
Alsaweed, W., & Aljebreen, S. (2024). Investigating the accuracy of ChatGPT as a writing error correction tool. International Journal of Computer-Assisted Language Learning and Teaching, 14(1), 1–18. https://doi.org/10.4018/ijcallt.364847.
5.
Alsofyani, A. H., & Barzanji, A. M. (2025). The effects of ChatGPT-generated feedback on Saudi EFL learners’ writing skills and perception at the tertiary level: A mixed-methods study. Journal of Educational Computing Research, 63(2), 431–463. https://doi.org/10.1177/07356331241307297.
6.
Asadi, M., Ebadi, S., & Mohammadi, L. (2025). The impact of integrating ChatGPT with teachers’ feedback on EFL writing skills. Thinking Skills and Creativity, 56, 101766. https://doi.org/10.1016/j.tsc.2025.101766.
7.
Barrot, J. S. (2023). Using ChatGPT for second language writing: Pitfalls and potentials. Assessing Writing, 57, 100745. https://doi.org/10.1016/j.asw.2023.100745.
8.
Bibi, Z., & Atta, A. (2024). The Role of ChatGPT as AI English Writing Assistant: A Study of Student’s Perceptions, Experiences, and Satisfaction. Annals of Human and Social Sciences, 5(I), 433–443. https://doi.org/10.35484/ahss.2024(5-i)39.
9.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley. https://doi.org/10.1002/9780470743386.
10.
Chan, S. T. S., Lo, N. P. K., & Wong, A. M. H. (2024). Enhancing university level English proficiency with generative AI: Empirical insights into automated feedback and learning outcomes. Contemporary Educational Technology, 16(4), ep541. https://doi.org/10.30935/cedtech/15607.
11.
Chen, C., & Gong, Y. F. (2025). The role of AI-assisted learning in academic writing: A mixed-methods study on Chinese as a second language students. Education Sciences, 15(2), 141. https://doi.org/10.3390/educsci15020141.
12.
Cingillioglu, I. (2023). Detecting AI-generated essays: The ChatGPT challenge. The International Journal of Information and Learning Technology, 40(3), 259–268. https://doi.org/10.1108/ijilt-03-2023-0043.
13.
Ding, L., Zou, D., & Kohnke, L. (2026). Enhancing L2 writing with AI: Effects of ChatGPT feedback and the role of student engagement. Artificial Intelligence in Language Education, 2, 103356. https://doi.org/10.29140/aile.2026.103356.
14.
Du, J. L., & Chen, Y. (2026). Exploring English for academic purposes learners’ engagement with ChatGPT-4 in academic writing revision: A case study from a private language institute in China. Frontiers in Artificial Intelligence, 9, 1802007. https://doi.org/10.3389/frai.2026.1802007.
15.
Du, J. L., Liu, Z., & Liao, N. (2026). Predicting higher education students’ continued intention and self-reported use of generative AI tools in academic work: An extended UTAUT mixed-methods study. Acta Psychologica, 269, 107471. https://doi.org/10.1016/j.actpsy.2026.107471.
16.
Du, J. L., & Kohnke, L. (2026). Repositioning teachers in the age of AI: A cross-disciplinary study of perceptions and practices in China. Teaching Education, 37(3), 281–305. https://doi.org/10.1080/10476210.2025.2590694.
17.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629.
18.
Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 20, 57. https://doi.org/10.1186/s41239-023-00425-2.
19.
Fan, N. (2023). Exploring the effects of automated written corrective feedback on EFL students’ writing quality: A mixed-methods study. SAGE Open, 13(2), 1–17. https://doi.org/10.1177/21582440231181296.
20.
Ferris, D. (2011). Treatment of error in second language student writing (2nd ed.). University of Michigan Press. https://doi.org/10.3998/mpub.2173290.
21.
Gayed, J. M., Carlon, M. K. J., Oriola, A. M., & Cross, J. S. (2022). Exploring an AI-based writing Assistant’s impact on English language learners. Computers and Education: Artificial Intelligence, 3, 100055. https://doi.org/10.1016/j.caeai.2022.100055.
22.
Ghafouri, M., Hassaskhah, J., & Mahdavi-Zafarghandi, A. (2024). From virtual assistant to writing mentor: Exploring the impact of a ChatGPT-based writing instruction protocol on EFL teachers’ self-efficacy and learners’ writing skill. Language Teaching Research. https://doi.org/10.1177/13621688241239764.
23.
Gibson, J. J. (1979). The ecological approach to visual perception. Houghton Mifflin.
24.
Godwin-Jones, R. (2022). Partnering with AI: Intelligent writing assistance and instructed language learning. Language Learning & Technology, 26(2), 5–24. https://doi.org/10.64152/10125/73474.
25.
Guo, K., Pan, M., Li, Y., & Lai, C. (2024). Effects of an AI-supported approach to peer feedback on university EFL students’ feedback quality and writing ability. The Internet and Higher Education, 63, 100962. https://doi.org/10.1016/j.iheduc.2024.100962.
26.
Harrer, M., Cuijpers, P., Furukawa, T. A., & Ebert, D. D. (2021). Doing meta-analysis with R: A hands-on guide. Chapman and Hall/CRC. https://doi.org/10.1201/9781003107347.
27.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Academic Press.
28.
Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of interventions (2nd ed.). Wiley. https://doi.org/10.1002/9781119536604.
29.
Hong, S., & Shin, Y. K. (2025). Effects of three levels of AI integration on second language academic writing: Evaluating restricted, guided, and free use of ChatGPT. System, 134, 103820. https://doi.org/10.1016/j.system.2025.103820.
30.
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M. P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M. C., Vedel, I., & Pluye, P. (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285–291. https://doi.org/10.3233/efi-180221.
31.
Hongxia, H., & Razali, A. B. (2025). Impact of ChatGPT on English academic writing ability and engagement of Chinese EFL undergraduates. International Journal of Instruction, 18(2), 323–346. https://doi.org/10.29333/iji.2025.18218a.
32.
Hu, W., Tian, J., & Li, Y. (2025). Enhancing student engagement in online collaborative writing through a generative AI-based conversational agent. The Internet and Higher Education, 65, 100979. https://doi.org/10.1016/j.iheduc.2024.100979.
33.
Hutchby, I. (2001). Technologies, texts and affordances. Sociology, 35(2), 441–456. https://doi.org/10.1017/S0038038501000219.
34.
Hyland, K., & Hyland, F. (2006). Feedback on second language students’ writing. Language Teaching, 39(2), 83–101. https://doi.org/10.1017/s0261444806003399.
35.
Hyland, K. (2019). Second language writing. Cambridge University Press. https://research-portal.uea.ac.uk/en/publications/second-language-writing/.
36.
Jin, F., Lin, C. H., & Lai, C. (2025). Modeling AI-assisted writing: How self-regulated learning influences writing outcomes. Computers in Human Behavior, 165, 108538. https://doi.org/10.1016/j.chb.2024.108538.
37.
Khany, R., & Barzan, P. (2026). Leveraging AI-powered writing assistants to enhance L2 writing proficiency: A mixed-methods study. AI and Tech in Behavioral and Social Sciences, 3(4), 1–14. https://doi.org/10.61838/kman.aitech.4287.
38.
Knapp, G., & Hartung, J. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22(17), 2693–2710. https://doi.org/10.1002/sim.1482.
39.
Koltovskaia, S., Rahmati, P., & Saeli, H. (2024). Graduate students’ use of ChatGPT for academic text revision: Behavioral, cognitive, and affective engagement. Journal of Second Language Writing, 65, 101130. https://doi.org/10.1016/j.jslw.2024.101130.
40.
Kurt, G., & Kurt, Y. (2024). Enhancing L2 writing skills: ChatGPT as an automated feedback tool. Journal of Information Technology Education: Research, 23, 024. https://doi.org/10.28945/5370.
41.
Kwon, S. K., Shin, D., & Lee, Y. (2023). The application of chatbot as an L2 writing practice tool. Language Learning & Technology, 27(1), 1–19. https://doi.org/10.64152/10125/73541.
42.
Lingard, L. (2023). Writing with ChatGPT: An Illustration of its Capacity, Limitations & Implications for Academic Writers. Perspectives on Medical Education, 12(1), 261–270. https://doi.org/10.5334/pme.1072.
43.
Liu, Z. M., Hwang, G. J., Chen, C. Q., Chen, X. D., & Ye, X. D. (2026). Integrating large language models into EFL writing instruction: Effects on performance, self-regulated learning strategies, and motivation. Computer Assisted Language Learning, 39(3), 466–490. https://doi.org/10.1080/09588221.2024.2389923.
44.
Lo, N., Chan, S., & Wong, A. (2025). Evaluating teacher, AI, and hybrid feedback in English language learning: Impact on student motivation, quality, and performance in Hong Kong. SAGE Open, 15(3). https://doi.org/10.1177/21582440251352907.
45.
Long, H. S. (2024). Exploring the use of ChatGPT as a tool for written corrective feedback in an EFL classroom. The Journal of AsiaTEFL, 21(2), 397–412. https://doi.org/10.18823/asiatefl.2024.21.2.8.397.
46.
Luo, M., Hu, X., & Zhong, C. (2025). The collaboration of AI and teacher in feedback provision and its impact on EFL learner’s argumentative writing. Education and Information Technologies, 30(12), 17695–17715. https://doi.org/10.1007/s10639-025-13488-7.
47.
Mahapatra, S. (2024). Impact of ChatGPT on ESL students’ academic writing skills: A mixed methods intervention study. Smart Learning Environments, 11, 9. https://doi.org/10.1186/s40561-024-00295-9.
48.
Meniado, J. C., Huyen, D. T. T., Panyadilokpong, N., & Lertkomolwit, P. (2024). Using ChatGPT for second language writing: Experiences and perceptions of EFL learners in Thailand and Vietnam. Computers and Education: Artificial Intelligence, 7, 100313. https://doi.org/10.1016/j.caeai.2024.100313.
49.
Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11(2), 364–386. https://doi.org/10.1177/1094428106291059.
50.
Munn, Z., Barker, T. H., Moola, S., Tufanaru, C., Stern, C., McArthur, A., Stephenson, M., & Aromataris, E. (2020). Methodological quality of case series studies: An introduction to the JBI critical appraisal tool. JBI Evidence Synthesis, 18(10), 2127–2133. https://doi.org/10.11124/jbisrir-d-19-00099.
51.
Nazari, N., Shabbir, M. S., & Setiawan, R. (2021). Application of Artificial Intelligence powered digital writing assistant in higher education: Randomized controlled trial. Heliyon, 7(5), e07014. https://doi.org/10.1016/j.heliyon.2021.e07014.
52.
Norris, J. M., & Ortega, L. (Eds.). (2006). Synthesizing research on language learning and teaching. John Benjamins. https://doi.org/10.1075/lllt.13.
53.
Ou, A. W., Khuder, B., Franzetti, S., & Negretti, R. (2024). Conceptualising and cultivating critical GAI literacy in doctoral academic writing. Journal of Second Language Writing, 66, 101156. https://doi.org/10.1016/j.jslw.2024.101156.
54.
Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. https://doi.org/10.1111/lang.12079.
55.
Plonsky, L. (2014). Study quality in quantitative L2 research (1990–2010): A methodological synthesis and call for reform. The Modern Language Journal, 98(1), 450–470. https://doi.org/10.1111/j.1540-4781.2014.12058.x.
56.
Polakova, P., & Ivenz, P. (2024). The impact of ChatGPT feedback on the development of EFL students’ writing skills. Cogent Education, 11(1), 2410101. https://doi.org/10.1080/2331186x.2024.2410101.
57.
Rad, H. S., Alipour, R., & Jafarpour, A. (2024). Using artificial intelligence to foster students’ writing feedback literacy, engagement, and outcome: A case of Wordtune application. Interactive Learning Environments, 32(9), 5020–5040. https://doi.org/10.1080/10494820.2023.2208170.
58.
Schmidt, R. W. (1990). The role of consciousness in second language learning. Applied Linguistics, 11(2), 129–158. https://doi.org/10.1093/applin/11.2.129.
59.
Shan, Z., Song, Z., Jiang, X., Chen, W., & Chen, L. (2025). Complementing but not replacing: Comparing the impacts of GPT-4 and native-speaker interaction on Chinese L2 writing outcomes. Behavioral Sciences, 15(4), 540. https://doi.org/10.3390/bs15040540.
60.
Shi, H., Chai, C. S., Zhou, S., & Aubrey, S. (2026). Comparing the effects of ChatGPT and automated writing evaluation on students’ writing and ideal L2 writing self. Computer Assisted Language Learning, 39(4), 1148–1175. https://doi.org/10.1080/09588221.2025.2454541.
61.
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H. Y., Corbett, M. S., Eldridge, S. M., Emberson, J. R., Hernán, M. A., Hopewell, S., Hróbjartsson, A., Junqueira, D. R., Jüni, P., Kirkham, J. J., Lasserson, T., Li, T., ... Higgins, J. P. T. (2019). RoB 2: A revised tool for assessing risk of bias in randomised trials. Bmj, 366, l4898. https://doi.org/10.1136/bmj.l4898.
62.
Stroud, R., & Du, J. (2025). Student acceptance of ChatGPT in higher education: A mixed-method study of attitudes, intentions, and concerns. Journal of Applied Learning & Teaching, 8(2), 35–48. https://doi.org/10.37074/jalt.2025.8.2.7.
63.
Su, Y., Lin, Y., & Lai, C. (2023). Collaborating with ChatGPT in argumentative writing classrooms. Assessing Writing, 57, 100752. https://doi.org/10.1016/j.asw.2023.100752.
64.
Teng, L. S., Deng, X., & Yang, J. (2026). Can GenAI-empowered feedback promote L2 learners’ self-regulation strategic behavior and writing performance? System, 138, 103970. https://doi.org/10.1016/j.system.2026.103970.
65.
Tran, T. T. T. (2025). Enhancing EFL writing revision practices: The impact of AI- and teacher-generated feedback and their sequences. Education Sciences, 15(2), 232. https://doi.org/10.3390/educsci15020232.
66.
Tsai, C. Y., Lin, Y. T., & Brown, I. K. (2024). Impacts of ChatGPT-assisted writing for EFL English majors: Feasibility and challenges. Education and Information Technologies, 29(17), 22427–22445. https://doi.org/10.1007/s10639-024-12722-y.
67.
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03.
68.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
69.
Wan, X., Liu, Q., & Adebayo, S. A. (2026). Explore Parents’ Views and Acceptance of Large Language Models Applied to Education. Journal of Educational Technology and Innovation, 8(2), 85–99. https://doi.org/10.61414/wbgfyr12.
70.
Wang, L., & Ren, B. (2024). Enhancing academic writing in a linguistics course with generative AI: An empirical study in a higher education institution in Hong Kong. Education Sciences, 14(12), 1329. https://doi.org/10.3390/educsci14121329.
71.
Wang, C., & Wang, Z. (2025). Investigating L2 writers’ critical AI literacy in AI-assisted writing: An APSE model. Journal of Second Language Writing, 67, 101187. https://doi.org/10.1016/j.jslw.2025.101187.
72.
Warschauer, M., Tseng, W., Yim, S., Webster, T., Jacob, S., Du, Q., & Tate, T. (2023). The affordances and contradictions of AI-generated text for writers of English as a second or foreign language. Journal of Second Language Writing, 62, 101071. https://doi.org/10.1016/j.jslw.2023.101071.
73.
Wei, P., Wang, X., & Dong, H. (2023). The impact of automated writing evaluation on second language writing skills of Chinese EFL learners: A randomized controlled trial. Frontiers in Psychology, 14, 1249991. https://doi.org/10.3389/fpsyg.2023.1249991.
74.
Wiboolyasarin, W., Wiboolyasarin, K., Suwanwihok, K., Jinowat, N., & Muenjanchoey, R. (2024). Synergizing collaborative writing and AI feedback: An investigation into enhancing L2 writing proficiency in wiki-based environments. Computers and Education: Artificial Intelligence, 6, 100228. https://doi.org/10.1016/j.caeai.2024.100228.
75.
Xu, J., & Zhang, S. (2022). Understanding AWE feedback and English writing of learners with different proficiency levels in an EFL classroom: A sociocultural perspective. The Asia-Pacific Education Researcher, 31(4), 357–367. https://doi.org/10.1007/s40299-021-00577-7.
76.
Yan, D. (2024). Comparing individual vs. collaborative processing of ChatGPT-generated feedback: Effects on L2 writing task improvement and learning. Language Learning & Technology, 28(1), 1–19. https://doi.org/10.64152/10125/73597.
77.
Yan, D. (2025). Feedback seeking abilities of L2 writers using ChatGPT: A mixed method multiple case study. Kybernetes, 54(7), 3757–3781. https://doi.org/10.1108/k-09-2023-1933.
78.
Yan, D., & Zhang, S. (2024). L2 writer engagement with automated written corrective feedback provided by ChatGPT: A mixed-method multiple case study. Humanities and Social Sciences Communications, 11, 1086. https://doi.org/10.1057/s41599-024-03543-y.
79.
Yang, W., & Lin, C. (2025). Translanguaging with generative AI in EFL writing: Students’ practices and perceptions. Journal of Second Language Writing, 67, 101181. https://doi.org/10.1016/j.jslw.2025.101181.
80.
Yeung, S. (2025). University students’ engagement with generative AI-supported automated writing evaluation (AWE) feedback. Journal of Second Language Writing, 68, 101203. https://doi.org/10.1016/j.jslw.2025.101203.
81.
Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–39). Academic Press. https://doi.org/10.1016/b978-012109890-2/50031-7.
82.
Zou, M., & Huang, L. (2024). The impact of ChatGPT on L2 writing and expected responses: Voice from doctoral students. Education and Information Technologies, 29, 13201–13219. https://doi.org/10.1007/s10639-023-12397-x.
83.
Zou, S., Guo, K., Wang, J., & Liu, Y. (2026). Investigating students’ uptake of teacher- and ChatGPT-generated feedback in EFL writing: A comparison study. Computer Assisted Language Learning, 39(4), 1062–1091. https://doi.org/10.1080/09588221.2024.2447279.

Scilight Press

Author Information

Abstract

Keywords

References

About Scilight

Journals

Publishing Policies

Contact Us