Cues For Speech Recognition Research Articles

Temporal Effects on Monaural Amplitude-Modulation Sensitivity in Ipsilateral, Contralateral and Bilateral Noise.

The amplitude modulations (AMs) in speech signals are useful cues for speech recognition. Several adaptation mechanisms may make the detection of AM in noisy backgrounds easier when the AM carrier is presented later rather than earlier in the noise. The aim of the present study was to characterize temporal adaptation to noise in AM detection. AM detection thresholds were measured for monaural (50ms, 1.5kHz) pure-tone carriers presented at the onset ('early' condition) and 300ms after the onset ('late' condition) of ipsilateral, contralateral, and bilateral (diotic) broadband noise, as well as in quiet. Thresholdswere 2-4dB better in the late than in the early condition for the three noise lateralities. The temporal effect held for carriers at equal sensation levels, confirming that it was not due to overshoot on carrier audibility. The temporal effect was larger for broadband than for low-band contralateral noises. Many aspects in the results were consistent with the noise activating the medial olivocochlear reflex (MOCR) and enhancing AM depth in the peripheral auditory response. Other aspects, however, indicate that central masking and adaptation unrelated to the MOCR also affect both carrier-tone and AM detection and are involved in the temporal effects.

Intelligibility Threshold for Cued Speech in H.264 Video Conference

Intelligibility Threshold for Cued Speech in H.264 Video Conference This paper discusses the cued speech recognition methods in videoconference. Cued speech is a specific gesture language that is used for communication between deaf people. We define the criteria for sentence intelligibility according to answers of testing subjects (deaf people). In our tests we use 30 sample videos coded by H.264 codec with various bit-rates and various speed of cued speech. Additionally, we define the criteria for consonant sign recognizability in single-handed finger alphabet (dactyl) analogically to acoustics. We use another 12 sample videos coded by H.264 codec with various bit-rates in four different video formats. To interpret the results we apply the standard scale for subjective video quality evaluation and the percentual evaluation of intelligibility as in acoustics. From the results we construct the minimum coded bit-rate recommendations for every spatial resolution.

A method for transcribing the manual components of Cued Speech

Designed to allow visual communication of speech signals, Cued Speech consists of discrete hand signals that are produced in synchrony with the visual mouth movements of speech. The purpose of this paper is to describe a method for transcribing these hand signals. Procedures are presented for identifying (1) the steady-state portion of the cue to be analyzed, (2) the cue’s handshape, and (3) the cue’s placement. Reliability is evaluated, using materials from 12 cuers that were transcribed on two separate occasions (either by the original rater or a second rater). Results show very good intra-rater and inter-rater reliability on average, which remained good across a variety of individual cuers, even when the cuer’s hand gestures were heavily coarticulated. Given its high reliability, this transcription method may be of benefit to applications that require systematic and quantitative analysis of Cued Speech production in various populations. In addition, some of the transcription principles from this method may be helpful in improving accuracy of automatic Cued Speech recognition systems.

Cued Speech automatic recognition in normal-hearing and deaf subjects

This article discusses the automatic recognition of Cued Speech in French based on hidden Markov models (HMMs). Cued Speech is a visual mode which, by using hand shapes in different positions and in combination with lip patterns of speech, makes all the sounds of a spoken language clearly understandable to deaf people. The aim of Cued Speech is to overcome the problems of lipreading and thus enable deaf children and adults to understand spoken language completely. In the current study, the authors demonstrate that visible gestures are as discriminant as audible orofacial gestures. Phoneme recognition and isolated word recognition experiments have been conducted using data from a normal-hearing cuer. The results obtained were very promising, and the study has been extended by applying the proposed methods to a deaf cuer. The achieved results have not shown any significant differences compared to automatic Cued Speech recognition in a normal-hearing subject. In automatic recognition of Cued Speech, lip shape and gesture recognition are required. Moreover, the integration of the two modalities is of great importance. In this study, lip shape component is fused with hand component to realize Cued Speech recognition. Using concatenative feature fusion and multi-stream HMM decision fusion, vowel recognition, consonant recognition, and isolated word recognition experiments have been conducted. For vowel recognition, an 87.6% vowel accuracy was obtained showing a 61.3% relative improvement compared to the sole use of lip shape parameters. In the case of consonant recognition, a 78.9% accuracy was obtained showing a 56% relative improvement compared to the use of lip shape only. In addition to vowel and consonant recognition, a complete phoneme recognition experiment using concatenated feature vectors and Gaussian mixture model (GMM) discrimination was conducted, obtaining a 74.4% phoneme accuracy. Isolated word recognition experiments in both normal-hearing and deaf subjects were also conducted providing a word accuracy of 94.9% and 89%, respectively. The obtained results were compared with those obtained using audio signal, and comparable accuracies were observed.

Lip Shape and Hand Position Fusion for Automatic Vowel Recognition in Cued Speech for French

Cued speech is a visual mode of communication that uses handshapes and placements in combination with the mouth movements of speech to make the phonemes of a spoken language look different from each other and clearly understandable to deaf and hearing-impaired people. The aim of cued speech is to overcome the problems of lip reading and thus enable deaf children and adults to wholly understand spoken language. Cued speech recognition requires hand gesture recognition and lip shape recognition, and also integration of the two components. This article presents hidden Markov model (HMM)-based vowel recognition as used in Cued Speech for French. Based on concatenative feature fusion and multistream HMM decision fusion, lip shape and hand position components were integrated into a single component, and automatic vowel recognition was realized. In the case of multistream HMM decision fusion, the obtained vowel classification accuracy using lip shape and hand position information was 87.6%, showing absolute improvement of 19.6% in comparison with a use restricted only to lip parameters.

Cantonese lexical tone recognition from frequency-specific temporal envelope and periodicity components in the same versus different noise band carriers

Temporal envelope and periodicity components (TEPCs) in the speech signal offer important cues for speech recognition. A recent study revealed that TEPCs extracted from the high-frequency regions contribute significantly to Cantonese lexical tone recognition at the monosyllabic word level (Yuen et al., 2006). The aim of the current study was to compare the contributions of TEPCs from different frequency regions for Cantonese lexical tone recognition in connected speech, using the same or different noise band carriers and a very limited number of frequency bands. Test stimuli were Cantonese disyllabic word minimal pairs differing from each other only in lexical tone. The test stimuli set exhausted all possible 15 lexical tone-pair contrasts (six lexical tones in total) for the first and the second syllables. The TEPC of the test items were extracted by full-wave rectification and low-pass filtering and were used to modulate a speech-spectrum-shaped noise carrier to create the test stimuli. The stimuli therefore contain only TEPCs but no fine structure components (FSCs) of the original signal. Multiple sets of stimuli were created with different combinations of TEPC-modulated frequency bands. Twenty normal-hearing subjects participated in the study. Even in a single-band condition, much robust lexical tone recognition was obtained with TEPC provided from the high frequency region than those from the low frequency region. This high frequency region TEPC advantage was consistent using noise band carriers from different frequency regions. The TEPCs in the high-frequency region are more important for Cantonese lexical tone recognition, when FSCs are not available. The results indicate a potential to improve speech recognition in tonal languages by preserving or enhancing the critical TEPCs via new signal-processing algorithms in hearing prosthesis. Copyright © 2009 John Wiley & Sons, Ltd.