Virtual grasps recognition using fusion of Leap Motion and force myography

Abstract

Hand gesture recognition is important for interactions under VR environment. Traditional vision-based approaches encounter occlusion problems, and thus, wearable devices could be an effective supplement. This study presents a hand grasps recognition method in virtual reality settings, by fusing signals acquired using force myography (FMG), a muscular activity-based hand gesture recognition method, and Leap Motion. We conducted an experiment where participants performed grasping of virtual objects with VR goggles on their head, an FMG band on their wrist, and a Leap Motion positioned either on the desk or on the goggles (two experimental settings). The FMG, Leap Motion, and fusion of both signals were used for training and testing a simple, but effective linear discriminant analysis classifier, as well as three other mainstream classification algorithms. The results showed that the fusion of both signals achieved a significant improvement in classification accuracy, compared to using Leap Motion alone in both experimental settings.