FunPianoAR: A Novel AR Application for Piano Learning Considering Paired Play Based on Multi-Marker Tracking

Abstract

This paper describes the design and implementation of a novel augmented reality application, referred as FunPianoAR, that aiming to activate the interests of user and improve the experience of piano learning. The FunPianoAR with a user-friendly interface considers the paired play to further reduce the difficulty of playing the piano for adult novices. For the paired play, one user plays the melody and the other plays the harmony part. The app is developed by using Android Studio 3.0 and artoolkitX 1.0, an open source augmented reality SDK, and installed on the AR smart glasses, Epson Moverio BT300. Due to the textureless features of real piano and registration precision, the application is implemented by using the fiducial marker tracking instead of the markerless recognition. Besides, we divided the piano keyboard into four zones and each zone used a separate marker for tracking to solve the limited field of view (FOV) to some extent. The virtual keys can be accurately superimposed on the piano keys by using multi-marker tracking. An evaluation was also conducted to compare the effects of two types of augmented reality information superimposed, i.e., the instant way without the hints for the next note to be played and the FunPianoAR with the hints for the coming notes to be played. The correctness rate was calculated and the time differences between the notes that two players should play at the same time were collected. The FunPianoAR shows more advantages over the instant way based on the evaluation results.