An initial exploration of semi-automated tutoring

Abstract

In this paper, we begin our process of incorporating an AI bot in an online chat tutoring setting as a support for the tutor. We explore how an AI bot could give suggestions for tutor messages, although the human tutor will control how to communicate with the student. Tutoring, an important dimension of networked learning, has long been seen as a beneficial approach to students’ learning. An AI bot has the potential to aid tutors in the tutoring process and contribute to the scalability. The present pilot study was conducted in the tutoring setting of the Math Coach program. In the program, teacher students aid students from upper primary school to upper secondary school in mathematics through an online text-based chat system. Llama2 was used as a large language model (LLM), fine-tuned for Swedish comprehension utilizing the Math Coach system's chat logs. Four coaches, teacher students at a technical Swedish university and active in the Math Coach program, were invited to interact with the AI bot and participate in a group discussion. The coaches interacted individually with the AI bot while the chat conversation was displayed on a monitor so all participants could discuss the interaction while it took place. A semi-structured interview approach was taken and the participants were also encouraged to 'think aloud' about their experience. In the discussions, the coaches expressed surprise by the AI's social aspect. They perceived the AI bot as friendly with a positive attitude and were especially surprised by its ability to correctly place appropriate emojis. The coaches agreed that the AI was able to ask both appropriate and helpful questions and share some good guidance for how to proceed in the problem-solving process. However, they felt that the AI bot was not able to offer sufficient mathematical guidance, oftentimes the AI bot was confidently wrong. It also wrote too long messages, which humans would typically separate into several chat messages, and did not wait for a response but instead moved too quickly towards the solution. Moving forward we plan to address the effects of improved prompts on the AI bot and continue finetuning the LLM. We will continue to conduct pilot studies and eventually conduct more large-scale empirical studies.