Design and Implementation of Gesture Music Composition System based on RealSense and Recurrent Neural Network

Abstract

Aiming at the problems of slow human-computer interaction and low accuracy of hand gesture recognition (HGR) in music composition systems such as human-computer interaction, gesture recognition, and algorithmic composition, this paper designs an HGR music composition system based on real-sense technology and recurrent neural network. The system uses an Intel RealSense 3D camera to shoot the user's hand and extract the information on the hand joint points with Smart Wearable Biosensors (SWBSs). The joint points are used to construct a three-dimensional model of the hand bone, and the joint points' equivalent distance and joint deflection are calculated by the joint point information to match the corresponding Curwen HGR. The matching music data is input into the recurrent neural network. GRU is combined with the Markov chain algorithm to complete the composition and avoid the phenomenon of gradient disappearance or gradient explosion. The HGR recognition function and composition algorithm are simulated and tested to verify the method's feasibility. Cohort verification investigations and efficacy evaluations of wearable biosensors are essential to support their clinical acceptability. The research results show that the HGR music composition system based on real-sense technology and recurrent neural networks can effectively identify the tester's HGRs and assist the tester in completing the composition using intelligent biosensors. The composition mode is close to the professional composition process.