Abstract
Numerous studies have investigated the effectiveness of audio-visual multimodal learning for speech enhancement (AVSE) tasks, seeking a solution that uses visual data as auxiliary and complementary input to reduce the noise of noisy speech signals. Recently, we proposed a lite audio-visual speech enhancement (LAVSE) algorithm for a car-driving scenario. Compared to conventional AVSE systems, LAVSE requires less online computation and to some extent solves the user privacy problem on facial data. In this study, we extend LAVSE to improve its ability to address three practical issues often encountered in implementing AVSE systems, namely, the additional cost of processing visual data, audio-visual asynchronization, and low-quality visual data. The proposed system is termed improved LAVSE (iLAVSE), which uses a convolutional recurrent neural network architecture as the core AVSE model. We evaluate iLAVSE on the Taiwan Mandarin speech with video dataset. Experimental results confirm that compared to conventional AVSE systems, iLAVSE can effectively overcome the aforementioned three practical issues and can improve enhancement performance. The results also confirm that iLAVSE is suitable for real-world scenarios, where high-quality audio-visual sensors may not always be available.
Highlights
S PEECH is the most natural and convenient means for human-human and human-machine communications
We extend the lite audio-visual speech enhancement (LAVSE) system to an improved LAVSE system, which is formed by a multimodal convolutional recurrent neural networks (RNNs) (CRNN) architecture in which the recurrent part is realized by implementing an long short-term memory (LSTM) layer
We proposed the improved LAVSE (iLAVSE) system, which aims to address three issues that may be encountered when developing practical audio-visual multimodal learning for speech enhancement (AVSE) systems, namely the high cost of processing visual data, audio-visual asynchronization, and low-quality visual data
Summary
S PEECH is the most natural and convenient means for human-human and human-machine communications. Various DL-based model structures, including deep denoising autoencoders [51], [52], fully connected neural networks [53], [54], [55], convolutional neural networks (CNNs) [56], [57], recurrent neural networks (RNNs), and long short-term memory (LSTM) [58], [59], [60], [61], [62], [63], have been used as the core model of an SE system and have been proven to provide better performance than traditional statistical and machine-learning methods Another well-known advantage of DL models is that they can flexibly fuse data from different domains [64], [65]. Based on the special design of model architecture and data augmentation, iLAVSE can effectively overcome the above three issues and provide more robust SE performance than LAVSE and several related SE methods.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
