Speech Cues Research Articles

Visual cues influence prosody perception among individuals with cochlear implants

Understanding a talker’s intention conveyed through prosody is essential for successful speech communication. Although visual speech cues are typically characterized by lip movements, there are also visual gestures including head tilts and eyebrow raises that signal prosody. Listeners with cochlear implants (CIs) may rely on these visual cues to solidify their prosody perception and guard against misinterpretations, particularly because of their poor perception of voice pitch. This study used audio-visual recordings of sentences in which a talker sometimes focused one word to signify a change in meaning. After characterizing acoustic and visual prosody cues in this stimulus set, we made prosodically mismatched variants of the stimuli by transplanting prosodically focused audio onto broad focus (i.e., prosodically unfocused) video, and vice versa. Visual influences on prosody perception were measured using acoustic analysis of participant reproductions in a vocal mimicry task. Results show reproduction of F0 contour—a key acoustic marker of prosodic focus—is influenced by perception of visual prosody cues. Specifically, visual cues indicating a focused word led to higher peak F0 in vocal reproductions of that word in the sentence. These results underscore the importance of multisensory information in supporting prosody perception for listeners with hearing loss.

Audiovisual speech perception in noise in younger and older bilinguals.

Speech perception in noise becomes increasingly difficult with age. Similarly, bilinguals often have difficulty with speech perception in noise in their second language (L2) due to less developed language knowledge in L2. Little is known about older bilinguals, who experience age-related sensory and cognitive changes but have extensive L2 experience. Furthermore, while audiovisual (AV) speech cues and supportive sentence context facilitate speech perception in noise in native listeners, much less is known for bilingual listeners, particularly older bilinguals. This study investigated how much young (n = 30) and older (n = 31) French-English/English-French bilinguals benefit from AV speech cues and sentence context in their first (L1) and second language during speech perception in noise. Participants completed the task in L1 and L2. Importantly, the level of background noise was individually calibrated for each participant to equate performance during the baseline task. The results suggest that there were no age differences in overall performance during the speech perception in noise task. However, older adults showed smaller language effects and smaller modality effects (i.e., comparing auditory-only speech perception to audiovisual speech perception) compared to younger adults. Furthermore, both young and older bilinguals benefited from AV speech cues and sentence context in both of their languages. However, during L2 speech perception in noise, visual speech cues were particularly important for young adults. In contrast, older adults show equal benefit from visual speech cues in L1 and L2. Taken together, these findings extend current models of AV speech perception to a second language in both younger and older adults. Furthermore, the findings suggest that bilinguals are likely to benefit from visual speech cues and supportive sentence context, although this benefit may differ slightly between younger and older bilinguals. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Processing of Visual Speech Cues in Speech-in-Noise Comprehension Depends on Working Memory Capacity and Enhances Neural Speech Tracking in Older Adults With Hearing Impairment.

Comprehending speech in noise (SiN) poses a challenge for older hearing-impaired listeners, requiring auditory and working memory resources. Visual speech cues provide additional sensory information supporting speech understanding, while the extent of such visual benefit is characterized by large variability, which might be accounted for by individual differences in working memory capacity (WMC). In the current study, we investigated behavioral and neurofunctional (i.e., neural speech tracking) correlates of auditory and audio-visual speech comprehension in babble noise and the associations with WMC. Healthy older adults with hearing impairment quantified by pure-tone hearing loss (threshold average: 31.85-57 dB, N = 67) listened to sentences in babble noise in audio-only, visual-only and audio-visual speech modality and performed a pattern matching and a comprehension task, while electroencephalography (EEG) was recorded. Behaviorally, no significant difference in task performance was observed across modalities. However, we did find a significant association between individual working memory capacity and task performance, suggesting a more complex interplay between audio-visual speech cues, working memory capacity and real-world listening tasks. Furthermore, we found that the visual speech presentation was accompanied by increased cortical tracking of the speech envelope, particularly in a right-hemispheric auditory topographical cluster. Post-hoc, we investigated the potential relationships between the behavioral performance and neural speech tracking but were not able to establish a significant association. Overall, our results show an increase in neurofunctional correlates of speech associated with congruent visual speech cues, specifically in a right auditory cluster, suggesting multisensory integration.

Fusing facial and speech cues for enhanced multimodal emotion recognition

Fusing facial and speech cues for enhanced multimodal emotion recognition

Multisensory effects of mask wearing on speech intelligibility and the benefit of multilingualism.

Due to the pandemic of the Covid-19 disease, it became common to wear masks on some public spaces. By covering mouth and nose, visual-related speech cues are greatly reduced, while the auditory signal is both distorted and attenuated. The present study aimed to analyze the multisensory effects of mask wearing on speech intelligibility and the differences in these effects between participants who spoke 1, 2 and 3 languages. The study consisted of the presentation of sentences from the SPIN test to 40 participants. Participants were asked to report the perceived sentences. There were four conditions: auditory with mask; audiovisual with mask; auditory without mask; audiovisual without mask. Two sessions were conducted, one week apart, each with the same stimuli but with a different signal-to-noise ratio. Results demonstrated that the use of the mask decreased speech intelligibility, both due to a decrease in the quality of auditory stimuli and due to the loss of visual information. Signal-to-noise ratio largely affects speech intelligibility and higher ratios are needed in mask-wearing conditions to obtain any degree of intelligibility. Those who speak more than one language are less affected by mask wearing, as are younger listeners. Wearing a facial mask reduces speech intelligibility, both due to visual and auditory factors. Older people and people who only speak one language are affected the most.

A Multimodal Late Fusion Framework for Physiological Sensor and Audio-Signal-Based Stress Detection: An Experimental Study and Public Dataset

Stress can be considered a mental/physiological reaction in conditions of high discomfort and challenging situations. The levels of stress can be reflected in both the physiological responses and speech signals of a person. Therefore the study of the fusion of the two modalities is of great interest. For this cause, public datasets are necessary so that the different proposed solutions can be comparable. In this work, a publicly available multimodal dataset for stress detection is introduced, including physiological signals and speech cues data. The physiological signals include electrocardiograph (ECG), respiration (RSP), and inertial measurement unit (IMU) sensors equipped in a smart vest. A data collection protocol was introduced to receive physiological and audio data based on alterations between well-known stressors and relaxation moments. Five subjects participated in the data collection, where both their physiological and audio signals were recorded by utilizing the developed smart vest and audio recording application. In addition, an analysis of the data and a decision-level fusion scheme is proposed. The analysis of physiological signals includes a massive feature extraction along with various fusion and feature selection methods. The audio analysis comprises a state-of-the-art feature extraction fed to a classifier to predict stress levels. Results from the analysis of audio and physiological signals are fused at a decision level for the final stress level detection, utilizing a machine learning algorithm. The whole framework was also tested in a real-life pilot scenario of disaster management, where users were acting as first responders while their stress was monitored in real time.

Supporting home language access using Cued Speech.

Supporting home language access using Cued Speech.

Exploring audiovisual speech perception in monolingual and bilingual children in Uzbekistan

This study aimed to investigate the development of audiovisual speech perception in monolingual Uzbek-speaking and bilingual Uzbek–Russian-speaking children, focusing on the impact of language experience on audiovisual speech perception and the role of visual phonetic (i.e., mouth movements corresponding to phonetic/lexical information) and temporal (i.e., timing of speech signals) cues. A total of 321 children aged 4 to 10 years in Tashkent, Uzbekistan, discriminated /ba/ and /da/ syllables across three conditions: auditory-only, audiovisual phonetic (i.e., sound accompanied by mouth movements), and audiovisual temporal (i.e., sound onset/offset accompanied by mouth opening/closing). Effects of modality (audiovisual phonetic, audiovisual temporal, or audio-only cues), age, group (monolingual or bilingual), and their interactions were tested using a Bayesian regression model. Overall, older participants performed better than younger participants. Participants performed better in the audiovisual phonetic modality compared with the auditory modality. However, no significant difference between monolingual and bilingual children was observed across all modalities. This finding stands in contrast to earlier studies. We attribute the contrasting findings of our study and the existing literature to the cross-linguistic similarity of the language pairs involved. When the languages spoken by bilinguals exhibit substantial linguistic similarity, there may be an increased necessity to disambiguate speech signals, leading to a greater reliance on audiovisual cues. The limited phonological similarity between Uzbek and Russian might have minimized bilinguals’ need to rely on visual speech cues, contributing to the lack of group differences in our study.

Improving speech command recognition through decision-level fusion of deep filtered speech cues

Improving speech command recognition through decision-level fusion of deep filtered speech cues

Role of Visual Speech Cues (Cued Speech) in Foreign Language Learning by Hearing School-Age Children

In this study, we aimed to determine the role of visual speech cues in the process of foreign ‎language learning by hearing school-age children. Our experiments used Cued Speech, a method ‎designed for people who are deaf or hard of hearing. We expected that the principles of the ‎method might also be beneficial for people with normal hearing because they may help distinguish ‎the sounds of foreign speech that are difficult to hear. This study mainly focused on the effects ‎of speech perception. We tested 126 Polish junior high school students (66 girls and 60 boys) ‎with a normal range of phonemic hearing and language aptitude. We envisaged that foreign ‎language learners using visual speech cues would achieve a higher score on a test of foreign ‎language than learners who had studied the language in the traditional manner. We also ‎formulated a hypothesis concerning the interaction of training type and training conditions on the ‎effectiveness of foreign language learning: that the difference in the effects of foreign language ‎learning between participants who received visual or executive training and typical training would ‎be more significant in the presence of auditory distractors than in their absence. We observed ‎interactions between conditions and types of training for speech sound identification. Under ‎conditions of auditory distraction, foreign language learners using Cued Speech scored ‎significantly higher than learners who had traditional training.‎

Surface and Contextual Linguistic Cues in Dialog Act Classification: A Cognitive Science View.

What role do linguistic cues on a surface and contextual level have in identifying the intention behind an utterance? Drawing on the wealth of studies and corpora from the computational task of dialog act classification, we studied this question from a cognitive science perspective. We first reviewed the role of linguistic cues in dialog act classification studies that evaluated model performance on three of the most commonly used English dialog act corpora. Findings show that frequency-based, machine learning, and deep learning methods all yield similar performance. Classification accuracies, moreover, generally do not explain which specific cues yield high performance. Using a cognitive science approach, in two analyses, we systematically investigated the role of cues in the surface structure of the utterance and cues of the surrounding context individually and combined. By comparing the explained variance, rather than the prediction accuracy of these cues in a logistic regression model, we found that (1) while surface and contextual linguistic cues can complement each other, surface linguistic cues form the backbone in human dialog act identification, (2) with word frequency statistics being particularly important for the dialog act, and (3) the similar trends across corpora, despite differences in the type of dialog, corpus setup, and dialog act tagset. The importance of surface linguistic cues in dialog act classification sheds light on how both computers and humans take advantage of these cues in speech act recognition.

Visual cues in sound change: A cross-modal perceptual account for the typological rarity of labial palatalization

Auditory perceptibility is widely recognized as a major contributing factor to listener-based sound change. Although speech perception is also known to be influenced by vision, the role of visual cues in facilitating or inhibiting sound change is not well understood. Palatalization of /tʲ/ and /kʲ/ to /tʃ/ is a typologically common change that is frequently attributed to misperception, while full palatalization of labial consonants is exceedingly rare. We test whether visual cues contribute to this asymmetry by way of the visual distinctiveness of labial closure. An audiovisual perception experiment was conducted in which 40 native Korean speakers were presented with synthetically altered tokens along 5-step continua from /pʰ/, /tʰ/, or /kʰ/ to /t?ɕ/ in audio-only, audiovisual-congruent, and audiovisual-incongruent conditions. Acoustic cues to place of articulation, F2 onset, and spectral frequency during aspiration, were manipulated. Results show that presence of the visual lip closure cue significantly inhibited perception of labials as palatals: the /pʰ/-/tɕʰ/ boundary was shifted toward /tɕʰ/ in the AV-congruent group, and labials were more likely to be perceived as palatals in the absence of lip closure. This finding suggests that visual speech cues may contribute to the typological asymmetry and that sound change can be modulated by visual cues.

Checkerboard and interrupted speech: Intelligibility contrasts related to factor-analysis-based frequency bandsa).

It has been shown that the intelligibility of checkerboard speech stimuli, in which speech signals were periodically interrupted in time and frequency, drastically varied according to the combination of the number of frequency bands (2-20) and segment duration (20-320 ms). However, the effects of the number of frequency bands between 4 and 20 and the frequency division parameters on intelligibility have been largely unknown. Here, we show that speech intelligibility was lowest in four-band checkerboard speech stimuli, except for the 320-ms segment duration. Then, temporally interrupted speech stimuli and eight-band checkerboard speech stimuli came in this order (N = 19 and 20). At the same time, U-shaped intelligibility curves were observed for four-band and possibly eight-band checkerboard speech stimuli. Furthermore, different parameters of frequency division resulted in small but significant intelligibility differences at the 160- and 320-ms segment duration in four-band checkerboard speech stimuli. These results suggest that factor-analysis-based four frequency bands, representing groups of critical bands correlating with each other in speech power fluctuations, work as speech cue channels essential for speech perception. Moreover, a probability summation model for perceptual units, consisting of a sub-unit process and a supra-unit process that receives outputs of the speech cue channels, may account for the U-shaped intelligibility curves.

Checkerboard and interrupted speech: Critical intelligibility differences observed in factor-analysis-based checkerboard speech stimuli

It has been shown that the intelligibility of checkerboard speech stimuli, in which speech signals were periodically interrupted in time and frequency, drastically varied according to the combination of the number of frequency bands (2–20) and segment duration (20–320 ms). However, the effects of the number of frequency bands between 4 and 20 and the ways of frequency divisions on intelligibility have been basically unknown. Here we show that 8-band checkerboard speech stimuli were more intelligible than temporally interrupted speech stimuli and 4-band checkerboard speech stimuli, irrespective of segment duration (n = 19 and 20). At the same time, U-shaped intelligibility curves were observed for 4-band and possibly 8-band checkerboard speech stimuli. Furthermore, when the frequency divisions were determined according to regular intervals on a critical bandwidth scale rather than factor analysis results of speech power fluctuations, intelligibility improved at the 160- and 320-ms segment duration. These results suggest that the factor-analysis-based frequency bands perform as speech cue channels that are essential for speech perception and that a probability summation model based on a perceptual unit of speech perception may account for the U-shaped intelligibility curves if the model is modified to include the speech cue channels.

Remote Augmented Reality Application: A Study on Cues and Behavioural Dimension

Remote augmented reality (AR) collaboration promotes an interactive way to present information to the user by conveying a message and instruction to the local and remote participants. Despite its advantages, it is found that due to the limited use of sensory modalities during the remote collaboration process, it can interrupt the transmission of information and interaction cues, by not conveying the right information in remote AR collaboration in which can affect focus, and responses between local and remote users. This study is intended to investigate the behavioural dimension of collaboration (collaborator’s behaviour) and cues involved between local and remote user for physical task. Six participants performed as local participants where they need to build a LEGO, while another 6 participants performed as remote participants that have a complete manual instruction. Participants were given maximum 60 minutes to complete the given task. The results shown that most of the time participants used gesture and speech cues to interact with each other. There are certain signals and keywords established by both participants to have mutual understanding in achieving desired goal. Moreover, it was shown that the task completed by using handsfree produce faster response.

Word Learning in Deaf Adults Who Use Cochlear Implants: The Role of Talker Variability and Attention to the Mouth.

Although cochlear implants (CIs) facilitate spoken language acquisition, many CI listeners experience difficulty learning new words. Studies have shown that highly variable stimulus input and audiovisual cues improve speech perception in CI listeners. However, less is known whether these two factors improve perception in a word learning context. Furthermore, few studies have examined how CI listeners direct their gaze to efficiently capture visual information available on a talker's face. The purpose of this study was two-fold: (1) to examine whether talker variability could improve word learning in CI listeners and (2) to examine how CI listeners direct their gaze while viewing a talker speak. Eighteen adults with CIs and 10 adults with normal hearing (NH) learned eight novel word-object pairs spoken by a single talker or six different talkers (multiple talkers). The word learning task comprised of nonsense words following the phonotactic rules of English. Learning was probed using a novel talker in a two-alternative forced-choice eye gaze task. Learners' eye movements to the mouth and the target object (accuracy) were tracked over time. Both groups performed near ceiling during the test phase, regardless of whether they learned from the same talker or different talkers. However, compared to listeners with NH, CI listeners directed their gaze significantly more to the talker's mouth while learning the words. Unlike NH listeners who can successfully learn words without focusing on the talker's mouth, CI listeners tended to direct their gaze to the talker's mouth, which may facilitate learning. This finding is consistent with the hypothesis that CI listeners use a visual processing strategy that efficiently captures redundant audiovisual speech cues available at the mouth. Due to ceiling effects, however, it is unclear whether talker variability facilitated word learning for adult CI listeners, an issue that should be addressed in future work using more difficult listening conditions.

Dialect and gender perception in relation to the intelligibility of low-pass and high-pass filtered spontaneous speecha).

Most cues to speech intelligibility are within a narrow frequency range, with its upper limit not exceeding 4 kHz. It is still unclear whether speaker-related (indexical) information is available past this limit or how speaker characteristics are distributed at frequencies within and outside the intelligibility range. Using low-pass and high-pass filtering, we examined the perceptual salience of dialect and gender cues in both intelligible and unintelligible speech. Setting the upper frequency limit at 11 kHz, spontaneously produced unique utterances (n = 400) from 40 speakers were high-pass filtered with frequency cutoffs from 0.7 to 5.56 kHz and presented to listeners for dialect and gender identification and intelligibility evaluation. The same material and experimental procedures were used to probe perception of low-pass filtered and unmodified speech with cutoffs from 0.5 to 1.1 kHz. Applying statistical signal detection theory analyses, we found that cues to gender were well preserved at low and high frequencies and did not depend on intelligibility, and the redundancy of gender cues at higher frequencies reduced response bias. Cues to dialect were relatively strong at low and high frequencies; however, most were in intelligible speech, modulated by a differential intelligibility advantage of male and female speakers at low and high frequencies.

The Role of Word Content, Sentence Information, and Vocoding for Voice Cue Perception.

For voice perception, two voice cues, the fundamental frequency (fo) and/or vocal tract length (VTL), seem to largely contribute to identification of voices and speaker characteristics. Acoustic content related to these voice cues is altered in cochlear implant transmitted speech, rendering voice perception difficult for the implant user. In everyday listening, there could be some facilitation from top-down compensatory mechanisms such as from use of linguistic content. Recently, we have shown a lexical content benefit on just-noticeable differences (JNDs) in VTL perception, which was not affected by vocoding. Whether this observed benefit relates to lexicality or phonemic content and whether additional sentence information can affect voice cue perception as well were investigated in this study. This study examined lexical benefit on VTL perception, by comparing words, time-reversed words, and nonwords, to investigate the contribution of lexical (words vs. nonwords) or phonetic (nonwords vs. reversed words) information. In addition, we investigated the effect of amount of speech (auditory) information on fo and VTL voice cue perception, by comparing words to sentences. In both experiments, nonvocoded and vocoded auditory stimuli were presented. The outcomes showed a replication of the detrimental effect reversed words have on VTL perception. Smaller JNDs were shown for stimuli containing lexical and/or phonemic information. Experiment 2 showed a benefit in processing full sentences compared to single words in both fo and VTL perception. In both experiments, there was an effect of vocoding, which only interacted with sentence information for fo. In addition to previous findings suggesting a lexical benefit, the current results show, more specifically, that lexical and phonemic information improves VTL perception. fo and VTL perception benefits from more sentence information compared to words. These results indicate that cochlear implant users may be able to partially compensate for voice cue perception difficulties by relying on the linguistic content and rich acoustic cues of everyday speech. https://doi.org/10.23641/asha.23796405.

The construct of attention in studies in l2 phonology between 2010 and 2021

The acquisition of a second language (L2) is a highly demanding cognitive task that, in terms of phonology, happens through the siege of the first language (L1). The cue-weighting, that is, the attention given to specific cues in the L2 speech and the ability to change the focus of attention to relevant cues is essential for L2 speech processing and tends to be affected by the patterns of the L1. The present study aimed to produce a systematic review of the studies concerning attention and L2 phonological knowledge. Research participants, attention and phonological assessment, and the main results of eighteen studies published between 2010 and 2021 were described and analyzed. The data gathered indicates that the bulk of the research carried out is of cross-sectional studies with attention control and perceptual discrimination tasks with L2 English adult learners. Most studies pointed out that attention has an important role in L2 phonological acquisition. At the beginning of acquisition, attention seems to be more important to L2 speech processing than for proficient L2 speakers when automatization takes place. Focusing, selecting, shifting, dividing, and sustaining attention as well as being flexible and having good inhibitory control have a special role in L2 phonological acquisition. The need for further investigation with controlled variables, other L1/L2 combinations, and multiple assessment instruments is evident.

Effects of noise and noise reduction on audiovisual speech perception in cochlear implant users: An ERP study

ObjectiveHearing with a cochlear implant (CI) is difficult in noisy environments, but the use of noise reduction algorithms, specifically ForwardFocus, can improve speech intelligibility. The current event-related potentials (ERP) study examined the electrophysiological correlates of this perceptual improvement. MethodsTen bimodal CI users performed a syllable-identification task in auditory and audiovisual conditions, with syllables presented from the front and stationary noise presented from the sides. Brainstorm was used for spatio-temporal evaluation of ERPs. ResultsCI users revealed an audiovisual benefit as reflected by shorter response times and greater activation in temporal and occipital regions at P2 latency. However, in auditory and audiovisual conditions, background noise hampered speech processing, leading to longer response times and delayed auditory-cortex-activation at N1 latency. Nevertheless, activating ForwardFocus resulted in shorter response times, reduced listening effort and enhanced superior-frontal-cortex-activation at P2 latency, particularly in audiovisual conditions. ConclusionsForwardFocus enhances speech intelligibility in audiovisual speech conditions by potentially allowing the reallocation of attentional resources to relevant auditory speech cues. SignificanceThis study shows for CI users that background noise and ForwardFocus differentially affect spatio-temporal cortical response patterns, both in auditory and audiovisual speech conditions.