Augmented Reality Maintenance Training with Intel Depth Camera

Abstract

Augmented Reality (AR) merges real and virtual objects in order to make a richer environment with additional useful virtual information. Producing AR maintenance training depends on two major points that will be Interactivity and Reusability. Recently Augmented Reality becomes the new technology revolution in software applications such as (entertainment, training, military, E-shops and games), and the new generation of hardware computers, mobiles, and consoles which can support the AR view. Always there is a limitation about user interactivity in using keyboard, mouse, and touch screens as input for virtual reality applications. Hand Gesture Recognition using depth cameras is an innovative solution to easily interact and manipulate virtual scene objects which are needed for computer users widely in entertainment or military applications. This research presents a solution for those areas and helps to design more realistic application with low cost and good efficiency. Using IntelRealSense (RS) Camera, which is a low cost hardware and very helpful as gesture and depth capture camera. Maintenance training was chosen as the case study for a reusable AR framework. This paper present a solution to two major problems: First, the calculation of occlusion between the real and the virtual objects including human hand. Second, problem will be to discuss object collision detection, to add more realism between the gesture recognition and the dynamic virtual scene. The proposed framework is designed to overcome or enhance the problems of developing augmented reality scenes that can be used to integrate between natural human activities and virtual world or real time computer graphics. So, maintenance training framework could make it possible for users to be trained, and helped with real-time instructions which can be followed to achieve the task goal without need to reference guide or separate catalog. The proposed framework succeeded in implementing maintenance training with gesture recognition stability approximately 90 percent with continues picking selection. Each part movements and speed of gesture action transition took less than a second with medium hand speed with approximately 1 second hand pose for gesture recognition.