Abstract
We present a new approach to the source separation problem in the case of multiple speech signals. The method is based on the use of automatic lipreading, the objective is to extract an acoustic speech signal from other acoustic signals by exploiting its coherence with the speaker′s lip movements. We consider the case of an additive stationary mixture of decorrelated sources, with no further assumptions on independence or non-Gaussian character. Firstly, we present a theoretical framework showing that it is indeed possible to separate a source when some of its spectral characteristics are provided to the system. Then we address the case of audio-visual sources. We show how, if a statistical model of the joint probability of visual and spectral audio input is learnt to quantify the audio-visual coherence, separation can be achieved by maximizing this probability. Finally, we present a number of separation results on a corpus of vowel-plosive-vowel sequences uttered by a single speaker, embedded in a mixture of other voices. We show that separation can be quite good for mixtures of 2, 3, and 5 sources. These results, while very preliminary, are encouraging, and are discussed in respect to their potential complementarity with traditional pure audio separation or enhancement techniques.
Highlights
There exists an intrinsic coherence and even a complementarity between audition and vision for speech perception [1]
We propose a new approach in the case of speech signal separation
The audio spectra envelopes were estimated by 20-order linear prediction (LP) models, which were calculated synchronously with the video parameters, on 32 ms frames, by using
Summary
There exists an intrinsic coherence and even a complementarity between audition and vision for speech perception [1]. Visual cues can compensate to a certain extent the deficiency of the auditory ones. This explains that the fusion of auditory and visual information meets a great success in several speech applications, mainly in speech recognition in noisy environments [3]. In a previous work [4], we tested a slightly different idea, we presented a prototype system which was able to exploit the visual input to enhance the audio signal corrupted by acoustic additive white noise. We propose a new approach in the case of speech signal separation. This approach, presented, is based on the use of the bimodality of speech and on the intrinsic coherence between audio and video speech.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
