Co-occurring Speech Research Articles

Intact speech-gesture integration in narrative recall by adults with moderate-severe traumatic brain injury

PurposeReal-world communication is situated in rich multimodal contexts, containing speech and gesture. Speakers often convey unique information in gesture that is not present in the speech signal (e.g., saying “He searched for a new recipe” while making a typing gesture). We examine the narrative retellings of participants with and without moderate-severe traumatic brain injury across three timepoints over two online Zoom sessions to investigate whether people with TBI can integrate information from co-occurring speech and gesture and if information from gesture persists across delays. Methods60 participants with TBI and 60 non-injured peers watched videos of a narrator telling four short stories. On key details, the narrator produced complementary gestures that conveyed unique information. Participants retold the stories at three timepoints: immediately after, 20-min later, and one-week later. We examined the words participants used when retelling these key details, coding them as a Speech Match (e.g., “He searched for a new recipe”), a Gesture Match (e.g., “He searched for a new recipe online), or Other (“He looked for a new recipe”). We also examined whether participants produced representative gestures themselves when retelling these details. ResultsDespite recalling fewer story details, participants with TBI were as likely as non-injured peers to report information from gesture in their narrative retellings. All participants were more likely to report information from gesture and produce representative gestures themselves one-week later compared to immediately after hearing the story. ConclusionWe demonstrated that speech-gesture integration is intact after TBI in narrative retellings. This finding has exciting implications for the utility of gesture to support comprehension and memory after TBI and expands our understanding of naturalistic multimodal language processing in this population.

Towards a corpus-based description of speech-gesture units of meaning

Abstract The theories and methods in corpus linguistics (CL) have had an impact on numerous areas in applied linguistics. However, the interface between CL and multimodal speech-gesture studies remains underexplored. One fundamental question is whether it is possible, and even appropriate, to apply the theories and paradigms established based on textual data to multimodal data. To explore this, we examine how CL can assist investigating lexico-grammatical patterns of speech co-occurring with a recurrent gesture (i.e. the circular gesture). Sinclair’s (1996) unit of meaning model is used to describe the co-gestural speech patterns. The study draws on a subset of the Nottingham Multimodal Corpus, in which 570 instances of circular gestures and their co-occurring speech are identified and analysed. We argue that Sinclair’s unit of meaning model can be extended to include speech-gesture patterns, and that those descriptions enable a more nuanced understanding of meaning in context.

Speech Perception Variability in Childhood Apraxia of Speech: Implications for Assessment and Intervention.

Beyond hallmark production deficits characterizing childhood apraxia of speech (CAS), largely attributed to disruption(s) in speech motor planning, children with CAS often present with co-occurring speech perception and language difficulties. Thus, careful consideration of the potential for speech perception difficulties to have cascading downstream effects on intervention responsiveness and real-life functioning for some children with CAS is highly important. The purpose of this tutorial was to consider the impact of speech perception abilities in children with CAS, which carries implications for caring for the needs of the whole child. This tutorial summarizes the current literature on speech perception and how it relates to speech production, language, and reading abilities for children with CAS. We include case illustrations that are adapted from real clinical scenarios illustrating how speech perception difficulties may impact some children with CAS and provide recommendations for incorporating speech perception into assessment and intervention practices. Although speech perception difficulties do not seem to be a core deficit of CAS, they are strongly linked to language difficulties, which are highly prevalent among children with CAS. Speech perception and language difficulties are also associated with reading difficulties and risk for lower academic achievement. Children with CAS who have co-occurring language difficulties likely also demonstrate speech perception deficits, which puts them at heightened risk for reading difficulties and struggles with academic achievement. Comprehensive assessment of children with CAS should address speech perception and production, language, and reading abilities, which carries important implications for multifaceted approaches to intervention.

Speech and language development and genotype-phenotype correlation in 49 individuals with KAT6A syndrome.

Pathogenic KAT6A variants cause syndromic neurodevelopmental disability. "Speech delay" is reported, yet none have examined specific speech and language features of KAT6A syndrome. Here we phenotype the communication profile of individuals with pathogenic KAT6A variants. Medical and communication data were acquired via standardized surveys and telehealth-assessment. Forty-nine individuals (25 females; aged 1;5-31;10) were recruited, most with truncating variants (44/49). Intellectual disability/developmental delay (42/45) was common, mostly moderate/severe, alongside concerns about vision (37/48), gastrointestinal function (33/48), and sleep (31/48). One-third (10/31) had a diagnosis of autism. Seventy-three percent (36/49) were minimally-verbal, relying on nonverbal behaviors to communicate. Verbal participants (13/49) displayed complex and co-occurring speech diagnoses regarding the perception/production of speech sounds, including phonological impairment (i.e., linguistic deficits) and speech apraxia (i.e., motor planning/programming deficits), which significantly impacted intelligibility. Receptive/expressive language and adaptive functioning were also severely impaired. Truncating variants in the last two exons of KAT6A were associated with poorer communication, daily-living skills, and socialization outcomes. In conclusion, severe communication difficulties are present in KAT6A syndrome, typically on a background of significant intellectual disability, vision, feeding and motor deficits, and autism in some. Most are minimally-verbal, with apparent contributions from underlying motor deficits and cognitive-linguistic impairment. Alternative/augmentative communication (AAC) approaches are required for many into adult life. Tailored AAC options should be fostered early, to accommodate the best communication outcomes.

Investigation into the effectiveness of electropalatography in treating persisting speech sound disorders in adolescents with co-occurring developmental language disorder

ABSTRACT This study aimed to assess the effectiveness of Electropalatography (EPG) intervention in targeting specific phonemes/words in seven adolescents aged 14:10–18:06 with co-occurring speech sound and language disorders. Progress on individualised targets versus controls was evaluated following intervention undertaken as part of the participants’ usual speech and language therapy provision. As a group, the participants showed significantly greater progress on their targets than controls, indicating that the EPG intervention was effective. However, performance varied between participants, targets and school terms. Factors that may have influenced the effectiveness of intervention include spending more time on targets and focusing on a specific phoneme. Overall, the results suggest EPG should be considered as an intervention approach for this client group, even in the late teenage years.

Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation

This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using augmented reality. The work presents a multimodal elicitation study conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). In this study time windows for gesture and speech multimodal interactions are developed using the start and stop times of gestures and speech as well as the stoke times for gestures. While gestures commonly precede speech by 81 ms we find that the stroke of the gesture is commonly within 10 ms of the start of speech. Indicating that the information content of a gesture and its co-occurring speech are well aligned to each other. Lastly, the trends across the most common proposals for each modality are examined. Showing that the disagreement between proposals is often caused by a variation of hand posture or syntax. Allowing us to present aliasing recommendations to increase the percentage of users' natural interactions captured by future multimodal interactive systems.

Speech sound errors of Mandarin-speaking preschool children with co-occurring speech sound disorder and developmental language disorder

ABSTRACT This study examined the occurrence and types of speech sound errors in Mandarin-speaking children and compared those with co-occurring developmental language disorder (DLD) and speech sound disorders (SSDs) to those with SSD only and those with no DLD or SSD. The participants were 64 four-year-old Mandarin-speaking children, including 20 age-matched children with co-occurring SSD and LI (SSD + DLD), 20 with SSD only, and 24 with no DLD or SSD (typical development [TD]). Speech samples from a list of 20 pictures of common objects and animals were elicited in a picture-naming task. One speech-language therapist transcribed and analysed consonant production accuracy and coded speech errors into the types of substitution, omission, distortion, and addition and phonological processes into typical and atypical sound changes. The interrater reliability of consonant production analysis was checked by another experienced speech-language pathologist. Children with co-occurring SSD and DLD demonstrated the least accuracy of speech production among the three groups of children. Children with co-occurring SSD and DLD showed more frequent use of substitutions, omissions, typical sound changes (i.e. stopping of affricates, stopping of fricatives, deaspiration), and atypical sound changes (i.e. initial consonant deletion, fronting, affrication of stops) than children with SSD only, while no significant difference between the SSD and TD groups was observed in omission error types and atypical sound changes. These results suggest that the disorders in speech sound production in children with SSD and DLD may be associated with underlying phonological–linguistic processing.

Seeing first person changes gesture but saying first person does not

Abstract Events with a motor action component (e.g., handling an object) tend to evoke gestures from the point of view of a character (character viewpoint, or CVPT) while events with a path component (moving through space) tend to evoke gestures from the point of view of an observer (observer viewpoint, or OVPT). Events that combine both components (e.g., rowing a boat across a lake) seem to evoke both types of gesture, but it is unclear why narrators use one or the other. We carry out two manipulations to explore whether gestural viewpoint can be manipulated. Participants read a series of stories and retold them in two conditions. In the image condition, story sentences were presented with images from either the actor’s perspective (actor version) or the observer’s perspective (observer version). In the linguistic condition, the same sentences were presented in either the second person (you…) or the third person point of view (h/she…). The second person led participants to use the first person (I) in retelling. Gestures produced during retelling were coded as CVPT or OVPT. Participants produced significantly more CVPT gestures after seeing images from the point of view of an actor, but the linguistic manipulation did not affect viewpoint in gesture. Neither manipulation affected overall gesture rate, or co-occurring speech. We relate these findings to frameworks in which motor action and mental imagery are linked to viewpoint in gesture.

Integrated Speech and Language Interventions

It is known that linguistic domains interact as they develop in parallel alongside developing speech production processes. For children with compromised linguistic systems, interactions across domains present challenges and opportunities when considering interventions. The frequently encountered population of children with co-occurring speech sound disorder (SSD) and expressive language impairment are the focus of this article. This population is at increased risk for long-term language and literacy difficulties that impact education, social participation, and vocational outcomes. Integrated interventions are reviewed for their efficacy; in these interventions there is a scheduled focus on both speech and language, or both domains are intentionally targeted. Findings from intervention comparison studies show that a variety of different integrated interventions were equally effective in producing significant gains in speech accuracy. Interventions also produced the greatest effects in areas that were explicitly targeted, although incidental effects were also achieved. Recommendations are offered for a continuum of care for preschoolers who early on display delays in both speech and language. Our small evidence base highlights the need for further development and testing of multifaceted interventions in routine practice, to achieve speech normalization, phonological awareness gains, and oral language foundations as close to school entry as possible.

New Guidelines for Assessing Neurogenic Stuttering

InroductionNeurogenic stuttering is an acquired disorder of speech fluency in adults resulting most commonly from traumatic brain injury (TBI) and stroke. Neurogenic stuttering frequently co-occurs with aphasia and dysarthria. It is important to consider what makes the core of neurogenic stuttering and which disfluencies are related to aphasia. MethodsWe examined neurogenic stuttering characteristics in five male patients with TBI. Speech fluency was assessed in three tasks. The new guideline for identifying stuttering and discriminating it from co-occurring speech and language deficiencies was created. ResultsWe found out that a) stuttering frequencies varied between different speech tasks, b) the most common stuttering type was repetition and c) the severity of stuttering increased linearly as the cognitive demands of the task increased and so did the types of stuttering. ConclusionMore research in the area of neurogenic stuttering is needed, especially for developing valid and reliable methods that take into account co-occurring speech and language deficiencies and communicative demands in various discourses.

Speech abilities in preschool children with speech sound disorder with and without co-occurring language impairment.

The authors compared preschool children with co-occurring speech sound disorder (SSD) and language impairment (LI) to children with SSD only in their numbers and types of speech sound errors. In this post hoc quasi-experimental study, independent samples t tests were used to compare the groups in the standard score from different tests of articulation/phonology, percent consonants correct, and the number of omission, substitution, distortion, typical, and atypical error patterns used in the production of different wordlists that had similar levels of phonetic and structural complexity. In comparison with children with SSD only, children with SSD and LI used similar numbers but different types of errors, including more omission patterns ( p < .001, d = 1.55) and fewer distortion patterns ( p = .022, d = 1.03). There were no significant differences in substitution, typical, and atypical error pattern use. Frequent omission error pattern use may reflect a more compromised linguistic system characterized by absent phonological representations for target sounds (see Shriberg et al., 2005). Research is required to examine the diagnostic potential of early frequent omission error pattern use in predicting later diagnoses of co-occurring SSD and LI and/or reading problems.

When Do Speakers Use Gestures to Specify Who Does What to Whom? The Role of Language Proficiency and Type of Gestures in Narratives

Previous research has found that iconic gestures (i.e., gestures that depict the actions, motions or shapes of entities) identify referents that are also lexically specified in the co-occurring speech produced by proficient speakers. This study examines whether concrete deictic gestures (i.e., gestures that point to physical entities) bear a different kind of relation to speech, and whether this relation is influenced by the language proficiency of the speakers. Two groups of speakers who had different levels of English proficiency were asked to retell a story in English. Their speech and gestures were transcribed and coded. Our findings showed that proficient speakers produced concrete deictic gestures for referents that were not specified in speech, and iconic gestures for referents that were specified in speech, suggesting that these two types of gestures bear different kinds of semantic relations with speech. In contrast, less proficient speakers produced concrete deictic gestures and iconic gestures whether or not referents were lexically specified in speech. Thus, both type of gesture and proficiency of speaker need to be considered when accounting for how gesture and speech are used in a narrative context.

The Hands of Time: Temporal gestures in English speakers

AbstractDo English speakers think about time the way they talk about it? In spoken English, time appears to flow along the sagittal axis (front/back): the future is ahead and the past is behind us. Here we show that when asked to gesture about past and future events deliberately, English speakers often use the sagittal axis, as language suggests they should. By contrast, when producing co-speech gestures spontaneously, they use the lateral axis (left/right) overwhelmingly more often, gesturing leftward for earlier times and rightward for later times. This left-right mapping of time is consistent with the flow of time on calendars and graphs in English-speaking cultures, but is completely absent from conventional spoken metaphors.English speakers gesture on the lateral axis even when they are using front/back metaphors in their co-occurring speech. This speech-gesture dissociation is not due to any lack of lexical or constructional resources to spatialize time laterally in language, nor to any lack of physical resources to spatialize time sagittally in gesture. We propose that when speakers are describing sequences of events, they often use neither the Moving Ego nor Moving Time perspectives. Rather, they adopt a “Moving Attention” perspective, which is grounded in patterns of interaction with cultural artifacts, not in patterns of interaction with the natural environment. We suggest possible pragmatic, kinematic, and mnemonic motivations for the use of a lateral mental timeline in gesture and in thought. Gestures reveal an implicit spatial conceptualization of time that cannot be inferred from language.

Altered integration of speech and gesture in children with autism spectrum disorders

The presence of gesture during speech has been shown to impact perception, comprehension, learning, and memory in normal adults and typically developing children. In neurotypical individuals, the impact of viewing co-speech gestures representing an object and/or action (i.e., iconic gesture) or speech rhythm (i.e., beat gesture) has also been observed at the neural level. Yet, despite growing evidence of delayed gesture development in children with autism spectrum disorders (ASD), few studies have examined how the brain processes multimodal communicative cues occurring during everyday communication in individuals with ASD. Here, we used a previously validated functional magnetic resonance imaging (fMRI) paradigm to examine the neural processing of co-speech beat gesture in children with ASD and matched controls. Consistent with prior observations in adults, typically developing children showed increased responses in right superior temporal gyrus and sulcus while listening to speech accompanied by beat gesture. Children with ASD, however, exhibited no significant modulatory effects in secondary auditory cortices for the presence of co-speech beat gesture. Rather, relative to their typically developing counterparts, children with ASD showed significantly greater activity in visual cortex while listening to speech accompanied by beat gesture. Importantly, the severity of their socio-communicative impairments correlated with activity in this region, such that the more impaired children demonstrated the greatest activity in visual areas while viewing co-speech beat gesture. These findings suggest that although the typically developing brain recognizes beat gesture as communicative and successfully integrates it with co-occurring speech, information from multiple sensory modalities is not effectively integrated during social communication in the autistic brain.

Stuttering treatment for a school-age child with Down syndrome: A descriptive case report

BackgroundLittle is known about optimal treatment approaches and stuttering treatment outcomes for children with Down syndrome. Aims and methodThe purpose of this study was to investigate outcomes for a child with Down syndrome who received a combination of fluency shaping therapy and parent delivered contingencies for normally fluent speech, prolonged speech, and stuttered speech. ResultsIn-clinic speech measures obtained at post-treatment and at 4 months follow-up reflected improvements in fluency of 89.0% and 98.6%, respectively. The participant's beyond-clinic follow-up sample reflected an improvement of 95.5%. Following treatment, the participant demonstrated improved self-confidence, self-esteem, and improved participation and functioning at school. ConclusionsFindings suggest that fluency shaping with parental contingencies may be a viable treatment approach to reduce stuttering in children with Down syndrome. Future research using an experimental research design is warranted.Educational objectives: Readers will be able to describe (a) prevalence estimates of stuttering in individuals with Down syndrome, (b) the main components of a fluency shaping program for a child with Down syndrome who stutters and has co-occurring speech and language delays, and (c) speech and parent-, teacher-, and self-report treatment outcomes.

Telerobotic Pointing Gestures Shape Human Spatial Cognition

This paper aimed to explore whether human beings can understand gestures produced by telepresence robots. If it were the case, they can derive meaning conveyed in telerobotic gestures when processing spatial information. We conducted two experiments over Skype in the present study. Participants were presented with a robotic interface that had arms, which were teleoperated by an experimenter. The robot could point to virtual locations that represented certain entities. In Experiment 1, the experimenter described spatial locations of fictitious objects sequentially in two conditions: speech condition (SO, verbal descriptions clearly indicated the spatial layout) and speech and gesture condition (SR, verbal descriptions were ambiguous but accompanied by robotic pointing gestures). Participants were then asked to recall the objects' spatial locations. We found that the number of spatial locations recalled in the SR condition was on par with that in the SO condition, suggesting that telerobotic pointing gestures compensated ambiguous speech during the process of spatial information. In Experiment 2, the experimenter described spatial locations non-sequentially in the SR and SO conditions. Surprisingly, the number of spatial locations recalled in the SR condition was even higher than that in the SO condition, suggesting that telerobotic pointing gestures were more powerful than speech in conveying spatial information when information was presented in an unpredictable order. The findings provide evidence that human beings are able to comprehend telerobotic gestures, and importantly, integrate these gestures with co-occurring speech. This work promotes engaging remote collaboration among humans through a robot intermediary.

Direct and Indirect Effects of Stimulating Phoneme Awareness vs. Other Linguistic Skills in Preschoolers With Co-occurring Speech and Language Impairments

Aim: The purpose of this study was to examine the effects of an integrated phoneme awareness/ speech intervention in comparison to an alternating speech/morphosyntax intervention for specific areas targeted by the different interventions, as well as the extent of indirect gains in nontargeted areas. Method: A total of 30 children with co-occurring speech sound disorder and language impairment, average age 4;5, participated in the study, 18 from the United States and 12 from New Zealand. Children from matched pairs were randomly assigned to the 2 proven efficacious treatments, which were delivered in 6-week blocks separated by a 6-week break. Phoneme awareness, speech sound production, and oral language outcome measures were collected pretreatment and after each intervention block. Results and Conclusions: Both intervention groups made statistically significant gains in all measures, with the exception of a morpheme measure only approaching significance. There were clear trends in favor of the specificity of the interventions suggesting increased sample size might have led to some significant intervention differences. Results further implicate the need for early intervention that integrates oral language and phoneme awareness/early literacy skills for children with multiple deficits.

Conversational gestures in autism spectrum disorders: Asynchrony but not decreased frequency

Conversational or "co-speech" gestures play an important role in communication, facilitating turntaking, providing visuospatial information, clarifying subtleties of emphasis, and other pragmatic cues. Consistent with other pragmatic language deficits, individuals with autism spectrum disorders (ASD) are said to produce fewer conversational gestures, as specified in many diagnostic measures. Surprisingly, while research shows fewer deictic gestures in young children with ASD, there is a little empirical evidence addressing other forms of gesture. The discrepancy between clinical and empirical observations may reflect impairments unrelated to frequency, such as gesture quality or integration with speech. Adolescents with high-functioning ASD (n = 15), matched on age, gender, and IQ to 15 typically developing (TD) adolescents, completed a narrative task to assess the spontaneous production of speech and gesture. Naïve observers rated the stories for communicative quality. Overall, the ASD group's stories were rated as less clear and engaging. Although utterance and gesture rates were comparable, the ASD group's gestures were less closely synchronized with the co-occurring speech, relative to control participants. This gesture-speech synchrony specifically impacted communicative quality across participants. Furthermore, while story ratings were associated with gesture count in TD adolescents, no such relationship was observed in adolescents with ASD, suggesting that gestures do not amplify communication in this population. Quality ratings were, however, correlated with ASD symptom severity scores, such that participants with fewer ASD symptoms were rated as telling higher quality stories. Implications of these findings are discussed in terms of communication and neuropsychological functioning in ASD.

Good and Bad in the Hands of Politicians: Spontaneous Gestures during Positive and Negative Speech

According to the body-specificity hypothesis, people with different bodily characteristics should form correspondingly different mental representations, even in highly abstract conceptual domains. In a previous test of this proposal, right- and left-handers were found to associate positive ideas like intelligence, attractiveness, and honesty with their dominant side and negative ideas with their non-dominant side. The goal of the present study was to determine whether 'body-specific' associations of space and valence can be observed beyond the laboratory in spontaneous behavior, and whether these implicit associations have visible consequences. We analyzed speech and gesture (3012 spoken clauses, 1747 gestures) from the final debates of the 2004 and 2008 US presidential elections, which involved two right-handers (Kerry, Bush) and two left-handers (Obama, McCain). Blind, independent coding of speech and gesture allowed objective hypothesis testing. Right- and left-handed candidates showed contrasting associations between gesture and speech. In both of the left-handed candidates, left-hand gestures were associated more strongly with positive-valence clauses and right-hand gestures with negative-valence clauses; the opposite pattern was found in both right-handed candidates. Speakers associate positive messages more strongly with dominant hand gestures and negative messages with non-dominant hand gestures, revealing a hidden link between action and emotion. This pattern cannot be explained by conventions in language or culture, which associate 'good' with 'right' but not with 'left'; rather, results support and extend the body-specificity hypothesis. Furthermore, results suggest that the hand speakers use to gesture may have unexpected (and probably unintended) communicative value, providing the listener with a subtle index of how the speaker feels about the content of the co-occurring speech.

Context-based Word Acquisition for Situated Dialogue in a Virtual World

To tackle the vocabulary problem in conversational systems, previous work has applied unsupervised learning approaches on co-occurring speech and eye gaze during interaction to automatically acquire new words. Although these approaches have shown promise, several issues related to human language behavior and human-machine conversation have not been addressed. First, psycholinguistic studies have shown certain temporal regularities between human eye movement and language production. While these regularities can potentially guide the acquisition process, they have not been incorporated in the previous unsupervised approaches. Second, conversational systems generally have an existing knowledge base about the domain and vocabulary. While the existing knowledge can potentially help bootstrap and constrain the acquired new words, it has not been incorporated in the previous models. Third, eye gaze could serve different functions in human-machine conversation. Some gaze streams may not be closely coupled with speech stream, and thus are potentially detrimental to word acquisition. Automated recognition of closely-coupled speech-gaze streams based on conversation context is important. To address these issues, we developed new approaches that incorporate user language behavior, domain knowledge, and conversation context in word acquisition. We evaluated these approaches in the context of situated dialogue in a virtual world. Our experimental results have shown that incorporating the above three types of contextual information significantly improves word acquisition performance.