The benefits and side effects of using ChatGPT in programming – a pilot study in the teaching and learning lab technology
Authors
Files
Abstract
Generative AI chatbots such as ChatGPT offer new possibilities for supporting learning processes in programming education, but their effectiveness depends on their didactic integration. In a pilot study, students developed an Arduino project with the help of ChatGPT. Both the programs created by the students and their logbooks on prompt usage were analyzed. The logbooks were coded using a literature-based category system of effective prompts. The results show that the prompts were mostly imprecise, hardly contextualized, and rarely developed iteratively. Students often adopted the generated program code without understanding its functional logic, which was reflected in difficulties in recognizing and correcting code errors. The findings suggest that programming based solely on ChatGPT does little to promote knowledge acquisition among novices and may even hinder it.
References
[1] Skalka, J., Drlik, M., Benko, L., Kapusta, J., Rodriguez del Pino, J. C., Smyrnova-Trybulska, E., Turcinek, P. (2021). Conceptual framework for programming skills development based on microlearning and automated source code evaluation in virtual learning environment. Sustainability, 13(6), 3293.
https://doi.org/10.3390/su13063293 DOI: https://doi.org/10.3390/su13063293
[2] Groothuijsen, S., van den Beemt, A., Remmers, J. C., & van Meeuwen, L. W. (2024). AI chatbots in programming education: Students’ use in a scientific computing course and consequences for learning. Computers and Education: Artificial Intelligence, 7, 100290.
https://doi.org/10.1016/j.caeai.2024.100290 DOI: https://doi.org/10.1016/j.caeai.2024.100290
[3] Sun, D., Boudouaia, A., Zhu, C., & Li, Y. (2024). Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study. International Journal of Educational Technology in Higher Education, 21(1), 14.
https://doi.org/10.1186/s41239-024-00446-5 DOI: https://doi.org/10.1186/s41239-024-00446-5
[4] Leifheit, L., Loefflad, D., Belschner, S., Beuttler, B., Winkelmann, J., Meurers, W. D., & Holz, H. (2024). KI im Unterricht: Entwicklung von Lehrveranstaltungen für Lehramtsstudierende der Sprach- und MINT-Fächer. Ludwigsburger Beiträge zur Medienpädagogik, 24, 1–19. https://doi.org/10.21240/lbzm/24/08 DOI: https://doi.org/10.21240/lbzm/24/08
[5] White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv.
https://doi.org/10.48550/arXiv.2302.11382
[6] Polverini, G., & Gregorcic, B. (2024). How understanding large language models can inform the use of ChatGPT in physics education. European Journal of Physics, 45(2), 025701. DOI: https://doi.org/10.1088/1361-6404/ad1420
https://doi.org/10.1088/1361-6404/ad1b3a
[7] Greco, C. M., & Tagarelli, A. (2024). Bringing order into the realm of Transformer-based language models for artificial intelligence and law. Artificial Intelligence and Law, 32(4), 863–1010. DOI: https://doi.org/10.1007/s10506-023-09374-7
https://doi.org/10.1007/s10506-023-09372-x
[8] Tassoti, S. (2024). Assessment of students’ use of generative artificial intelligence: Prompting strategies and prompt engineering in chemistry education. Journal of Chemical Education, 101(6), 2475–2482. DOI: https://doi.org/10.1021/acs.jchemed.4c00212
https://doi.org/10.1021/acs.jchemed.3c01069
[9] Malik, S. I., Ashfque, M. W., Tawafak, R. M., Al-Farsi, G., Usmani, N. A., & Khudayer, B. H. (2022). A chatbot to facilitate student learning in a Programming 1 course: A gendered analysis. International Journal of Virtual and Personal Learning Environments, 12(1), 1–20.
https://doi.org/10.4018/IJVPLE.310007 DOI: https://doi.org/10.4018/IJVPLE.310007
[10] Yılmaz, R., & Yılmaz, F. G. K. (2023). Augmented intelligence in programming learning: Examining student views on the use of ChatGPT for programming learning. Computers in Human Behavior: Artificial Humans, 1(2), 100005.
https://doi.org/10.1016/j.chbah.2023.100005 DOI: https://doi.org/10.1016/j.chbah.2023.100005
[11] Wang, T., Díaz, D. V., Brown, C., & Chen, Y. (2023). Exploring the role of AI assistants in computer science education: Methods, implications, and instructor perspectives. In 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (pp. 92–102). IEEE.
https://doi.org/10.1109/VL-HCC57772.2023.00018 DOI: https://doi.org/10.1109/VL-HCC57772.2023.00018
[12] Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783.
https://doi.org/10.3390/app13095783 DOI: https://doi.org/10.3390/app13095783
[13] Chen, E., Huang, R., Chen, H. S., Tseng, Y. H., & Li, L. Y. (2023). GPTutor: A ChatGPT-powered programming tool for code explanation. In International Conference on Artificial Intelligence in Education (pp. 321–327). Springer Nature Switzerland.
https://doi.org/10.1007/978-3-031-36336-8_50 DOI: https://doi.org/10.1007/978-3-031-36336-8_50
[14] Samek, W., & Schmid, U., et al. (2025). Nachvollziehbare KI: Erklären, für wen, was und wofür (Whitepaper der Plattform Lernende Systeme). Plattform Lernende Systeme / acatech.
https://doi.org/10.48669/pls_2025-2
[15] Beyerer, J., Kirchner, E., André, E., Behnke, S., Bloss, G., Dzaack, J., Egloffstein, T., Griepentrog, H. W., Gustmann, M., Hafner, V. V., Koert, D., Nüchter, A., Seyler, J., Straube, S., Tchouchenkov, I., von Stryk, O., Wedler, A., Wolf-Ostermann, K., & Zimmermann, M. (2025). KI in der Robotik: Flexible und anpassbare Systeme durch interaktives Lernen (Whitepaper der Plattform Lernende Systeme). Plattform Lernende Systeme / acatech.
https://doi.org/10.48669/pls_2025-1
[16] Stoyanova, D., Stoyanova-Petrova, S., & Mileva, N. (2025). Exploring students’ and teachers’ perceptions about using ChatGPT in programming education. International Journal of Engineering Pedagogy, 15(2). DOI: https://doi.org/10.3991/ijep.v15i2.50607
[17] Sagar, P. (2024). Exploring the use of ChatGPT and the prompting framework as a self-learning aid for Arduino coding & circuit building for artists and designers (Doctoral dissertation, Massachusetts Institute of Technology). MIT Libraries.
https://hdl.handle.net/1721.1/151432
[18] Ekin, S. (2023). Prompt engineering for ChatGPT: A quick guide to techniques, tips, and best practices. TechRxiv. DOI: https://doi.org/10.36227/techrxiv.22683919.v2
https://doi.org/10.36227/techrxiv.22683919.v1 DOI: https://doi.org/10.36227/techrxiv.22683919.v1
[19] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS 35) (pp. 24824–24837). DOI: https://doi.org/10.52202/068431-1800
[20] Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
https://doi.org/10.1177/001316446002000104 DOI: https://doi.org/10.1177/001316446002000104
[21] Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
https://doi.org/10.2307/2529310 DOI: https://doi.org/10.2307/2529310
[22] Röhner, G. (2016). Bildungsstandards Informatik für die Sekundarstufe II (Januar 2016). Gesellschaft für Informatik e. V.
https://doi.org/10.18420/rec2016_057
[23] Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., & Lakhani, K. R. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Harvard Business School Technology & Operations Mgt. Unit Working Paper, (24-013). DOI: https://doi.org/10.2139/ssrn.4573321
Downloads
Details
License
Copyright (c) 2026 Igor Gideon, Nico Link
This work is licensed under a Creative Commons Attribution 4.0 International License.