HAR-time: human action recognition with time factor analysis on worker operating time

Abstract

ABSTRACT This research proposed a data analysis framework applying human action recognition (HAR) with filtering processing to recognize human workers’ actions and further calculate the operating time of each action. When applying HAR to recognize the worker’s operation actions on a manufacturing site, the repeated and reciprocating actions easily cause misrecognition. Without the separation of actions in advance, the evaluation of HAR in real-world manufacturing is different from the evaluation based on the open data in the literature. To resolve the practical issues of applying HAR on a manufacturing site, first, the spatial-temporal skeleton coordinates of human joints were generated as the input of the skeleton-based spatial-temporal graph convolutional neural network (ST-GCN) for training the recognition model. Once the human action is recognized in each frame, the filtering processing considering the reference correction and accumulative moving mode (AMM) was proposed to improve the recognition accuracy. The real-world case study of the forging industry was conducted, and the empirical result shows the proposed framework was able to detect human workers’ repeated actions with 89% accuracy. Also, the time analysis based on HAR-Time recognition can correctly detect the associated time features that have highly correlated to the low quality of the product.