Exploring the Role of ChatGPT as a Peer Coach for Developing Research Proposals: Feedback Quality, Prompts, and Student Reflection

Abstract

This paper describes the results of an archival desk-study that analyzed worksheets produced by four students using ChatGPT as a coach. ChatGPT is a Generative Pre-Trained Large Language model that can write comprehensively in various languages and styles. It was discovered that it could pass university level physics exams and perform at the level of a third-year medical student. Fourth-year students in Information Technology are required to produce a 3000-to-5000-word research report as part of the requirements for graduation, and their first meeting of the year consists of a workshop following the GROW (Goal, Reality, Opportunities, Will) coaching process. Logistical considerations in this predominantly e-learning based course prevented such teamwork. The two-month-old ChatGPT, however, presented an opportunity to determine the extent to which a chatbot could be used as a peer coach. This paper reports on the outcome of an exercise that was given to students as an introduction to their research methodology course. It was found that well-designed prompts were essential and that students should be encouraged to reflect on their interaction with artificial intelligence. Three clear issues emerged from the study: the quality of feedback given by the chatbot, the value of the prompt, and the importance of student reflection. The chatbot, ChatGPT, displays the traditional computer characteristic of garbage in, garbage out and gives the feedback it is programmed to give. The dialogue shows how a student can build up mutual rapport with the chatbot when they adapt their responses to the feedback provided. The critical reflection in shows that it is still the user who should oversee the process and evaluate the feedback. It is recommended that novice research students be given some training in reacting to feedback, developing useful prompts, and engaging in meaningful reflection. Future research would include developing a reflective coach.