Fine-Grained Action Recognition in Assembly Work Scenes by Drawing Attention to the Hands

Abstract

The analysis of assembly working scenes is an important tool for improving work efficiency and detecting worker error. Many factories choose to analyze videos of employees working by human observation. It is more efficient, however, to approach this as an action segmentation task, but this is difficult due to the fine-grained actions. In this paper, we focus on featuring the workers hands in action segmentation to describe the detailed moves in assembly work scenes and utilize the tool or product information that the worker is using. We propose two methods to draw attention to the hand from each image in the video: the first is by cutting out the workers hand image and extracting features, and the second is by using an attention module specialized to extract hand features by the training of a regression network. For both methods we combine different types of features—image features and pose features—which are both important information in fine-grained actions. Due to the lack of action datasets that focus on assembly work scenes, we create a new assembly work dataset for our experiments, and the results of those experiments show that our method is effective for analyzing fine-grained action segmentation tasks.