Multi-sensory assessment for hand pattern recognition

Abstract

This study presents and assesses a multi-sensory hand pattern recognizer. We experimentally evaluated multiple features from the selected sensors, detected the most significant ones, and set the best combination possible to reach optimal hand shape detection. multilayer perceptron (MLP) and Decision Tree (DT) classifiers were used, and we used these two classification techniques to ensure that the results were not related to the classification technique. Leap Motion (LM) controllers and a VIVE Pro Eye camera were used to detect hand keypoints, and those last sensors are vision-based. We integrated this with a MYO armband, which streams eight EMG channels and inertial measurement unit data. All the sensors’ data were synchronized and streamed in real-time to be treated and fused. We evaluated our integration on the classification of the American Sign Language (ASL). We worked with seven participants, and we detected the best features subset. The overall results were satisfactory, wherein we reached an average classification of 98.31% with the optimal subset.