Hand Gesture Recognition for Controlling Game Objects Using Two-Stream Faster Region Convolutional Neural Networks Methods

Abstract

Hand gesture recognition is part of Human-Computer Interaction as a place for the user interface to be presented. Hand gestures have symbols that can be interpreted as commands or messages and can be used to control objects in games. The Faster R-CNN method can classify hand patterns based on specific regions but does not contain hand movement information for each frame in the video data. The data segmentation process with lighting, position, and configuration of hand gesture classification also significantly affects accuracy in hand gesture recognition. The purpose of this research is to separate the background of the hand gesture object and to be able to control the game object through the method applied. This research has built a hand gesture recognition system using the Two-Stream Faster R-CNN method in dividing spatial and temporal video data information by region to produce good accuracy. Data obtained from the camera by video processing using dense optical flow and frame selection, resulting in a sequence of spatial and temporal images. The resulting output can directly classify five hand gesture classes and control game objects. The results of this study indicate that the proposed method has the highest accuracy on spatial data at step 7500 with mAP values of 92.51% and temporal data at step 6500 with mAP values of 90.46%. IoU testing also affects the detection results with a minimum standard of 0.7 to 0.9 on the IoU coefficient value to avoid detection errors.