Human hand motion prediction based on feature grouping and deep learning: Pipe skid maintenance example

Abstract

Human-robot collaboration has gained popularity in various construction applications. The key to a successful human-robot collaboration at the same construction workplace is the delicate algorithm for predicting human motions to strengthen the robot's situational awareness, i.e., robot-human awareness. Most existing approaches have focused on predicting human motions based on repetitive patterns of human behaviors in well-defined task contexts, such as specific object picking tasks, for a relatively short period of time. These methods can hardly capture the “pattern inconsistency” of human actions, i.e., the differences across people in terms of motion features and even for the same person at different time points of the task. This paper proposes an analytical pipeline that segments and clusters the human inconsistent behaviors into different pattern groups and builds separate human motion pattern prediction models correspondingly. The proposed method, Human Motion Feature Grouping and Prediction (HMFGP), quantifies the spatiotemporal relationship between gaze focus and hand movement trajectories, segments the raw data based on the detected gaze-hand relationship pattern changes, and clusters the matched gaze-hand data segments into several pattern groups based on the pattern similarity of the gaze-hand relationships. Then a time series Deep Learning method is used to predict hand motions based on gaze focus trajectories for each of the pattern groups. The gaze and hand motion data of a human subject experiment (n = 120) for pipe skid maintenance was used to test the prediction performance of HMFGP. The result shows that HMFGP can significantly improve the accuracy of human hand motion prediction and help quantity different patterns of human motions for specific analyses.