Patient Assistance System Based on Hand Gesture Recognition

Abstract

We propose a two-stage hand gesture recognition architecture to support a patient assistance system. Some medical conditions limit mobility, and the patient must rely on medical staff to meet their needs. In such cases, a phone or intercom is not convenient to call for help. A vision-based system operated by changing the orientation of fingers can be used to send specific messages without making arm movements. However, vision-based hand gesture recognition is hindered by occlusion, background clutter, and variations in illumination. Therefore, we developed a two-stage architecture: The first stage produces a saliency map to simplify recognition and the second stage performs classification. A novel combined loss function optimizes the saliency detection model and makes the saliency map more precise. An adaptive kernel-based channel attention layer is used to emphasize salient features. The proposed architecture achieved precise saliency detection on four benchmark datasets and high-accuracy recognition on two. We designed an interface for patients to send specific messages to the medical staff using hand gestures. The interface help patients request assistance and connect with medical staff without leaving the bed or involving a third party.