A vision-based hand hygiene monitoring approach using self-attention convolutional neural network

Abstract

Automatic hand hygiene monitoring is an effective and necessary practice in healthcare. The main objective is to automate the quality control of the hand hygiene process by recognizing handwashing actions. This paper proposes a new computer vision-based hand hygiene monitoring approach, with the aim to recognize handwashing actions from the input videos. The proposed approach consists of the following three modules. First, in the feature extraction module, a backbone convolutional neural network (CNN) model (i.e., a ResNet model) is used to extract features from each frame of the input video. Second, in the feature aggregation module, the self-attention mechanism is introduced to form six masked self-attention blocks that are stacked to aggregate features from multiple frames. Third, in the feature classification module, the self-attentive aggregated feature representation is used for handwashing action recognition. The proposed approach outperforms a few state-of-the-art handwashing action recognition approaches. The proposed approach is able to perform effective and accurate handwashing action recognition for automatic hand hygiene monitoring, as verified by three benchmark datasets.