Enhancement of Human Sensory and Motor Skills using Collaboration of Eye - Ear Focus and Body - Arm - Hand Gesture in No-Touch 3D Interactive Aircraft Game

Abstract

Nowadays, human sensory and motor skills enhancement is very important to be alert from the effect of the Society 5.0 era. We propose the novel approach to enhance human sensory and motor skills using collaboration of senses focus and body parts gesture in no-touch3D interactive aircraft game. A player interacts with a virtual display using gestures of his body, arm, hand, or collaboration of them as a controller to engage in a 3D game. This game is based on an aircraft education theme. By sweeping the hand to the left or right, a player can select one of many different aircraft models according to its specification and performance are shown on the screen based on the estimation of his current skill. The mission is to control aircraft avoiding the random balls attack to prevent a collision. Hand and arm are positioned above a leap motion sensor and in front of a camera. The body, arm, and hand gesture directions are to forward or backward from the screen to the right or left, and also to up or down freely. This system is developed by using Unity 3D, OpenCV, and Python. Many skills are possible to be improved, e.g., motivation, discipline, safety awareness, endurance management, strategy, precision, intelligence, creativity, reliability, toughness, and vision. This project has been tested in the Air Force Museum of Dirgantara Mandala involving more than 1000 visitors as testing participants, and they all feel satisfied. The score is calculated by time in second. The results show endurance time is 32% (<60 s), 31% (<120 s), 25% (<160 s), and 12 % (180 s) of the total300 seconds of the game