Gesture-controlled interactive musical game to practice hand therapy exercises and learn rhythm and melodic structures

Abstract

This work presents an interactive musical game for health using MIDI files and the Leap Motion controller to detect gesture input from the users' hands and fingers. The musical notes of each song determine when specific hand gestures should be performed, creating physical activities that stimulate the hand motor development, as well as the learning of basic principles of rhythm and melodic structure of songs. According to the performed gestures and the musical notes played, a virtual puppet comes to life and moves in realtime, serving as an additional vehicle for stimulating the motor skill activities of users. Usability testing was conducted and the results show that participants (children and adults) successfully completed the hand therapy tasks and found the game entertaining. In addition, it was identified that the adults (with a preexisting repetitive strain injury condition) felt some physical discomfort during the exercises and had a higher count of gestures performed incorrectly. Furthermore, the difference of gesture execution according to the chosen level of difficulty stimulated participants to perceive which notes were the most important in the songs' sections.