Real-time full-body motion reconstruction and recognition for off-the-shelf VR devices

Abstract

Almost all VR multi-person applications have the requirement of reconstructing the whole-body motions in real-time in order to create deeper immersion. In fact, if we can ensure that the motion we reconstruct is natural, we can improve upon the comfort and number of wearing sensors at the cost of reconstruction precision because all users in VR are blindfolded. In this paper, we introduce a novel real-time motion reconstruction and recognition method in VR only using the positions and orientations of user's head and two hands. To reconstruct natural motions according to rare sensors, we divide the whole body into the upper body and the lower body. The upper body reconstruction algorithm based on inverse kinematics which is more accurate and the lower body based on animation blending which only needs a small number of prepared animations. Meanwhile, a natural action recognition algorithm based on neural network will run in need to detect the target motions we have trained. We show that our method can reconstruct various full-body motions, such as walking in any direction, jogging, jumping, crouching and turning. Our method has the ability to reconstruct natural human motions which can be used in almost all VR multi-person interactive applications. The configuration of sensors we attached on the head and hands is very popular in VR well-known devices, so the method can be easily integrated into off-the-shelf VR devices.