Solving the cocktail party problem using Multi-modal Hearing Assistive Technology Prototype

Abstract

Hearing loss is a major global health problem, affecting over 1.5 billion people. According to estimations by the World Health Organization, 83% of those who could benefit from hearing assistive devices do not use them. The limited adoption of hearing aids can be attributed to the suboptimal performance in acoustically challenging environments, such as cocktail parties, where there are multiple competing speakers and noise sources leading to poor speech intelligibility for hearing impaired listeners. This work presents a first-of-its-kind real-time, multi-modal speech enhancement system that can effectively enhance speech in challenging real noisy environments. The system exploits both audio and visual cues to isolate the target speaker's voice from interfering background noises. The system is developed using deep neural networks and is integrated with the Open Master Hearing Aid (openMHA) platform. The integrated prototype was evaluated in a cocktail party setting to isolate the target speaker's voice from interfering speakers, music, and other non-speech noise sources and was able to significantly improve the speech intelligibility of hearing impaired listeners. This work has the potential to improve the quality of life for hearing impaired listeners by facilitating effective communication in cocktail party environments.