Abstract
Abstract. The human body posture is rich with dynamic information that can be captured by algorithms, and many applications rely on this type of data (e.g., action recognition, people re-identification, human-computer interaction, industrial robotics). The recent development of smart cameras and affordable red-green-blue-depth (RGB-D) sensors has enabled cost-efficient estimation and tracking of human body posture. However, the reliability of single sensors is often insufficient due to occlusion problems, field-of-view limitations, and the limited measurement distances of the RGB-depth sensors. Furthermore, a large-scale real-time response is often required in certain applications, such as physical rehabilitation, where human actions must be detected and monitored over time, or in industries where human motion is monitored to maintain predictable movement flow in a shared workspace. Large-scale markerless motion-capture systems have therefore received extensive research attention in recent years.In this paper, we propose a real-time photogrammetric system that incorporates multithreading and a graphic process unit (GPU)-accelerated solution for extracting 3D human body dynamics in real-time. The system includes a stereo camera with preliminary calibration, from which left-view and right-view frames are loaded. Then, a dense image-matching algorithm is married with GPU acceleration to generate a real-time disparity map, which is further extended to a 3D map array obtained by photogrammetric processing based on the camera orientation parameters. The 3D body features are acquired from 2D body skeletons extracted from regional multi-person pose estimation (RMPE) and the corresponding 3D coordinates of each joint in the 3D map array. These 3D body features are then extracted and visualised in real-time by multithreading, from which human movement dynamics (e.g., moving speed, knee pressure angle) are derived. The results reveal that the process rate (pose frame-rate) can be 20 fps (frames per second) or above in our experiments (using two NVIDIA 2080Ti and two 12-core CPUs) depending on the GPU exploited by the detector, and the monitoring distance can reach 15 m with a geometric accuracy better than 1% of the distance.This real-time photogrammetric system is an effective real-time solution to monitor 3D human body dynamics. It uses low-cost RGB stereo cameras controlled by consumer GPU-enabled computers, and no other specialised hardware is required. This system has great potential for applications such as motion tracking, 3D body information extraction and human dynamics monitoring.
Highlights
Human body dynamics and posture evaluation have been an intensive research area for decades, in areas such as facial feature point-recognition algorithms (Ranjan et al, 2017; Xiong and De la Torre, 2013) and single- or multiple-person gesture recognition (Ghidoni and Munaro, 2017; Zanfir et al, 2013)
Accelerated advances in graphic processing unit (GPU) technology and the advent of multithreading-capable CPUs have recently led to the popularity of deep learning approaches, as exemplified by algorithms for real-time human posture evaluation, such as mask regional-based convolutional neural network (R-CNN) (Abdulla, 2017), OpenPose (Cao et al, 2018) and regional multi-person pose estimation (RMPE) (Fang et al, 2017)
To improve the running frame rate and efficiency of real-time 3D human body keypoint-detection and posture estimation in a largescale real-time response, here we describe a novel real-time photogrammetric system that incorporates multithreading and GPU acceleration
Summary
Human body dynamics and posture evaluation have been an intensive research area for decades, in areas such as facial feature point-recognition algorithms (Ranjan et al, 2017; Xiong and De la Torre, 2013) and single- or multiple-person gesture recognition (Ghidoni and Munaro, 2017; Zanfir et al, 2013). Accelerated advances in graphic processing unit (GPU) technology and the advent of multithreading-capable CPUs have recently led to the popularity of deep learning approaches, as exemplified by algorithms for real-time human posture evaluation, such as mask regional-based convolutional neural network (R-CNN) (Abdulla, 2017), OpenPose (Cao et al, 2018) and regional multi-person pose estimation (RMPE) (Fang et al, 2017) These deep learning-based object-detection and pose-evaluation algorithms can accurately obtain the 2D keypoints of human posture. As the 3D body skeleton information can be applied to human-movement monitoring and tracking, this system can simultaneously obtain the distance, direction and speed information of human body movement for various applications
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
