Accurate and real-time human-joint-position estimation for a patient-transfer robot using a two-level convolutional neutral network

Abstract

Human-joint-position estimation is crucial for patient-transfer robots. However, high accuracy and real-time property are difficult to achieve simultaneously. To tackle the problem, we develop a new convolutional neural network (CNN), containing two levels of subnetworks, to fuse the information in color and depth images. The first-level subnetwork generates two-dimensional (2D) human joint positions from a color image by the part-affinity-fields method. The second-level subnetwork estimates 3D human-joint positions from 2D ones and corresponding depth images. Here, strong feature-extraction function of the CNN may suppress the negative effect caused by invalid information in depth images. Meanwhile, all the estimations are implemented with the 2D CNNs, which may cause higher time-efficiency than 3D ones (mostly used in previous studies). To assess the validity, first we employed the CNN to estimate human joint positions, and obtained the accuracy and speed of respectively 90.3% and 210 ms (implemented with an affordable processing unit). Then we applied the CNN to a dual-arm nursing-care robot and found that the accuracy and processing speed satisfied the requirements in practical usage; these validated the effectiveness of our proposal and provided a new approach to generate 3D-human-joint positions through information fusion of color and depth images.