Sound Event Detection and Haptic Vibration Based Home Monitoring Assistant System for the Deaf and Hard-of-Hearing

Abstract

Acoustic signals contain a significant amount of information generated by sound sources. Unfortunately, deaf and hard-of-hearing people cannot access this information. Therefore, an assistive technology is required to help people with hearing loss. In this paper, we present a home monitoring assistant system based on sound event detection and sound-to-haptic conversion for the deaf and hard-of-hearing. The system detects the sounds erated in the home environment, converts the detected sound into text and haptic vibration, and provides them to the deaf and hard-of-hearing. The proposed approach is mainly composed of four modules, including signal estimation, reliable sensor channel selection, sound event detection, and conversion of sound into haptic vibration. During signal estimation, lost packets are recovered to improve the signal quality. Next, reliable channels are selected using a multi-channel cross-correlation coefficient to improve the computational efficiency for distant sound event detection. Finally, the sounds of the selected two channels are used for environmental sound event detection based on bidirectional gated recurrent neural networks and for sound-to-haptic effect conversion using kernel-based source separation. Experiments show that the proposed approach achieves superior performances compared to the baseline.