Privacy-Preserving Visual Cues Communication for Hearing-Impaired People Using Deep Learning

Abstract

Translate Article

Non-verbal communication is a crucial element of human interaction, serving as a powerful tool for expressing emotions, establishing connections, and understanding others beyond verbal language. The recognition of non-verbal cues has held significant importance in recent years, particularly for individuals with disabilities like deafness or muteness. This recognition plays a crucial role in enabling effective communication for these groups. However, existing camera-based systems for detecting non-verbal cues have drawbacks, including privacy concerns, difficulties in varying lighting conditions, the need for complex training, and operational range limitations. In this study, we employ contactless sensing technology to detect non-verbal cues such as Good Idea, Thinking, Worried, Normal, Shocked, and OK. This proof-of-concept study demonstrates the efficacy of this privacy-preserving system. The data was first transformed into spectrograms and then processed using deep learning models like ResNet50 and VGG16, achieving remarkable classification accuracy, notably 95.83% using ResNet50.