Impersonating Chatbots in a Code Review Exercise to Teach Software Engineering Best Practices

Abstract

Over the past decade, the use of chatbots for educational purposes has gained considerable traction. A similar trend has been observed in social coding platforms, where automated agents support software developers with tasks such as performing code reviews. While incorporating code reviews and social coding platforms into software engineering education has been found to be beneficial, challenges such as steep learning curves and privacy considerations are barriers to their adoption. Furthermore, no study has addressed the role chatbots play in supporting code reviews as a pedagogical tool. To help address this gap, we developed an online learning application that simulates the code review features available on social coding platforms and allows instructors to interact with students using chatbot identities. We then embedded this application within a lesson on software engineering best practices and conducted a controlled in-class experiment. This experiment examined the effect that explaining content via chatbot identities had on three aspects: (i) students’ perceived usability of the lesson, (ii) their engagement with the code review process, and (iii) their learning gains. While our findings show that it is feasible to simulate the code review process within an online learning platform and achieve good usability, our quantitative analysis did not yield significant differences across treatment conditions for any of the aspects considered. Nevertheless, our qualitative results suggest that students expect explicit feedback when performing this type of exercise and could thus benefit from automated replies provided by an interactive chatbot. We propose to build on our current findings to further explore this line of research in future work.