Abstract
We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fully-convolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control---thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e., it works for outdoor scenes, community videos, and low quality commodity RGB cameras.
Highlights
Optical skeletal motion capture of humans is widely used in applications such as character animation for movies and games, sports and biomechanics, and medicine
We experimentally show that this makes ours the first singleRGB method usable for similar real-time 3D applications—so far only feasible with RGB-D input—such as game character control or immersive first person virtual reality (VR)
We propose a 3D pose estimation approach that leverages a novel fully-convolutional convolutional neural network (CNN) formulation to predict 2D and 3D pose jointly
Summary
Optical skeletal motion capture of humans is widely used in applications such as character animation for movies and games, sports and biomechanics, and medicine. To overcome the usability constraints imposed by commercial systems requiring marker suits [Menache 2000], researchers developed marker-less motion capture methods that estimate motion in more general scenes using multi-view. The swell in popularity of applications such as realtime motion-driven 3D game character control, self-immersion in 3D virtual and augmented reality, and human–computer interaction, has led to new real-time full-body motion estimation techniques using only a single, easy to install, depth camera, such as the Microsoft Kinect [Microsoft Corporation 2010, 2013, 2015]. RGB-D cameras often fail in general outdoor scenes (due to sunlight interference), are bulkier, have higher power consumption, have lower resolution and limited range, and are not as widely and cheaply available as color cameras
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
