Mixed reality strategies for piano education

Abstract

Head-mounted mixed reality interfaces, in which a user perceives a seamless blend of real, virtual or remote content, have great potential in a wide range of educational contexts. In this paper, we explore the use of mixed reality (MR) in piano education. We begin with a review of previous examples of virtual and mixed reality for piano teaching and learning, identifying four major categories of functionality: remote teaching, learning to read musical notation, providing alternative notation systems to Western musical notation, and enhancing users’ understanding and experience of music. Following this review, we present an application designed to demonstrate the use of MR for facilitating remote student-teacher piano lessons. Additionally, hand-tracking enables real-time, bi-directional visualization of both the student’s and teacher’s hand movements while playing, which is a crucial communication channel in piano instruction. We also present the Piano Theory Hub, an interactive area for independent learning and practice designed to compliment remote instructions. The Piano Theory Hub uses MR to provide spatial in situ explanations and exercises on notes, intervals, scales, and chords. User studies were conducted to evaluate both remote and solo aspects of the application. The first study with 10 participants revealed a strong sense of immersion and co-presence with the teacher during remote lessons. Trials also revealed that, in addition to virtual hands, visual highlighting of key-presses was found to improve visualisation of the remote users’ play. The second study, with 15 participants found the Piano Theory Hub to be helpful for beginners and some intermediate players, with feedback suggesting improvements in user experience and highlighting the limitations of replacing traditional piano teachers. Our experiments demonstrate that MR can be successfully employed for the following strategies for piano education: hand interaction coaching, flexible virtual hand and piano visibility, augmented feedback including key highlighting, gamified learning elements, and flexible teacher positioning options. Overall, the findings suggest that mixed reality holds promise as an effective tool for remote piano learning and music education, offering immersive and engaging learning experiences.