Spoken Language Communication Research Articles

Linguistic analysis of human-computer interaction

This article reviews recent literature investigating speech variation in production and comprehension during spoken language communication between humans and devices. Human speech patterns toward voice-AI presents a test to our scientific understanding about speech communication and language use. First, work exploring how human-AI interactions are similar to, or different from, human-human interactions in the realm of speech variation is reviewed. In particular, we focus on studies examining how users adapt their speech when resolving linguistic misunderstandings by computers and when accommodating their speech toward devices. Next, we consider work that investigates how top-down factors in the interaction can influence users’ linguistic interpretations of speech produced by technological agents and how the ways in which speech is generated (via text-to-speech synthesis, TTS) and recognized (using automatic speech recognition technology, ASR) has an effect on communication. Throughout this review, we aim to bridge both HCI frameworks and theoretical linguistic models accounting for variation in human speech. We also highlight findings in this growing area that can provide insight to the cognitive and social representations underlying linguistic communication more broadly. Additionally, we touch on the implications of this line of work for addressing major societal issues in speech technology.

Test of Prosody via Syllable Emphasis ("TOPsy"): Psychometric Validation of a Brief Scalable Test of Lexical Stress Perception.

Prosody perception is fundamental to spoken language communication as it supports comprehension, pragmatics, morphosyntactic parsing of speech streams, and phonological awareness. A particular aspect of prosody: perceptual sensitivity to speech rhythm patterns in words (i.e., lexical stress sensitivity), is also a robust predictor of reading skills, though it has received much less attention than phonological awareness in the literature. Given the importance of prosody and reading in educational outcomes, reliable and valid tools are needed to conduct large-scale health and genetic investigations of individual differences in prosody, as groundwork for investigating the biological underpinnings of the relationship between prosody and reading. Motivated by this need, we present the Test of Prosody via Syllable Emphasis (“TOPsy”) and highlight its merits as a phenotyping tool to measure lexical stress sensitivity in as little as 10 min, in scalable internet-based cohorts. In this 28-item speech rhythm perception test [modeled after the stress identification test from Wade-Woolley (2016)], participants listen to multi-syllabic spoken words and are asked to identify lexical stress patterns. Psychometric analyses in a large internet-based sample shows excellent reliability, and predictive validity for self-reported difficulties with speech-language, reading, and musical beat synchronization. Further, items loaded onto two distinct factors corresponding to initially stressed vs. non-initially stressed words. These results are consistent with previous reports that speech rhythm perception abilities correlate with musical rhythm sensitivity and speech-language/reading skills, and are implicated in reading disorders (e.g., dyslexia). We conclude that TOPsy can serve as a useful tool for studying prosodic perception at large scales in a variety of different settings, and importantly can act as a validated brief phenotype for future investigations of the genetic architecture of prosodic perception, and its relationship to educational outcomes.

Articulatory, acoustic, and prosodic accommodation in a cooperative maze navigation task.

This study uses a maze navigation task in conjunction with a quasi-scripted, prosodically controlled speech task to examine acoustic and articulatory accommodation in pairs of interacting speakers. The experiment uses a dual electromagnetic articulography set-up to collect synchronized acoustic and articulatory kinematic data from two facing speakers simultaneously. We measure the members of a dyad individually before they interact, while they are interacting in a cooperative task, and again individually after they interact. The design is ideally suited to measure speech convergence, divergence, and persistence effects during and after speaker interaction. This study specifically examines how convergence and divergence effects during a dyadic interaction may be related to prosodically salient positions, such as preceding a phrase boundary. The findings of accommodation in fine-grained prosodic measures illuminate our understanding of how the realization of linguistic phrasal structure is coordinated across interacting speakers. Our findings on individual speaker variability and the time course of accommodation provide novel evidence for accommodation at the level of cognitively specified motor control of individual articulatory gestures. Taken together, these results have implications for understanding the cognitive control of interactional behavior in spoken language communication.

Audio–visual integration during speech perception in prelingually deafened Japanese children revealed by the McGurk effect

ObjectiveTo investigate the McGurk effect in profoundly deafened Japanese children with cochlear implants (CI) and in normal-hearing children. This was done to identify how children with profound deafness using CI established audiovisual integration during the speech acquisition period. MethodsTwenty-four prelingually deafened children with CI and 12 age-matched normal-hearing children participated in this study. Responses to audiovisual stimuli were compared between deafened and normal-hearing controls. Additionally, responses of the children with CI younger than 6 years of age were compared with those of the children with CI at least 6 years of age at the time of the test. ResultsResponses to stimuli combining auditory labials and visual non-labials were significantly different between deafened children with CI and normal-hearing controls (p<0.05). Additionally, the McGurk effect tended to be more induced in deafened children older than 6 years of age than in their younger counterparts. ConclusionsThe McGurk effect was more significantly induced in prelingually deafened Japanese children with CI than in normal-hearing, age-matched Japanese children. Despite having good speech-perception skills and auditory input through their CI, from early childhood, deafened children may use more visual information in speech perception than normal-hearing children. As children using CI need to communicate based on insufficient speech signals coded by CI, additional activities of higher-order brain function may be necessary to compensate for the incomplete auditory input. This study provided information on the influence of deafness on the development of audiovisual integration related to speech, which could contribute to our further understanding of the strategies used in spoken language communication by prelingually deafened children.

How much imitation is there in a shadowing task?

Phonetic imitation, also called phonetic convergence, is currently at the heart of numerous investigations since it can inform us on both the nature of lexical representations and the link between production and perception processes in spoken language communication. A task that has been largely used to study phonetic imitation is the shadowing task, in which participants merely listen to and repeat isolated words. In this study, we examined the extent to which the phonetic convergence effect found when participants shadow auditory tokens, is an imitation of the speaker. We thus compared the phonetic convergence effect observed in a shadowing task to that observed when participants were explicitly instructed to imitate the productions they were exposed to. Although the phonetic convergence effect was greater when participants intentionally imitated the speaker's productions, shadowing and imitation instructions led to the same degree of convergence in a post-exposure task. Hence, the convergence effect found in a shadowing task and that found in an imitation task seem to share a general mechanism which is automatic and which taps into the long-term representations of the words in memory. At a more theoretical level, our results reinforce the claim that detailed auditory traces associated with perceived words are stored in memory and are later used for production.

Audiologists and Speech-Language Pathologists: Making Critical Cross-Disciplinary Connections For Quality Care in Early Hearing Detection and Intervention

Widespread realization of newborn hearing screening has made it possible to routinely identify hearing loss shortly after birth, expanding opportunities for children born with permanent hearing loss. For children to reach their full potential, high-quality comprehensive services need to be provided in a timely manner. Because the roles of the audiologist and speech-language pathologist vary significantly from family to family in an American Sign Language approach, this article focuses primarily on the roles these professionals serve within a listening and spoken language communication approach. An overview of components of quality assessment and intervention for audiology and speech-language pathology are discussed, as are the benefits and opportunities of interdisciplinary collaboration. Newborn hearing screenings, advanced hearing technology, and early education have the potential to affect the lives of children with hearing loss and their families; however, successful families and children rely on quality, collaborative intervention from their service providers. Together, speech-language pathologists and audiologists can better understand a child’s responses to sound, more effectively set hearing technology to maximize access to sound, and support parents in their ability to help their children reach their full potential.

Evaluation of 2-way Iraqi Arabic–English speech translation systems using automated metrics

The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program ( http://1.usa.gov/transtac ) faced many challenges in applying automated measures of translation quality to Iraqi Arabic---English speech translation dialogues. Features of speech data in general and of Iraqi Arabic data in particular undermine basic assumptions of automated measures that depend on matching system outputs to reference translations. These features are described along with the challenges they present for evaluating machine translation quality using automated metrics. We show that scores for translation into Iraqi Arabic exhibit higher correlations with human judgments when they are computed from normalized system outputs and reference translations. Orthographic normalization, lexical normalization, and operations involving light stemming resulted in higher correlations with human judgments.

Evaluation methodology and metrics employed to assess the TRANSTAC two-way, speech-to-speech translation systems

One of the most difficult challenges that military personnel face when operating in foreign countries is clear and successful communication with the local population. To address this issue, the Defense Advanced Research Projects Agency (DARPA) is funding academic institutions and industrial organizations through the Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program to develop practical machine translation systems. The goal of the TRANSTAC program is to demonstrate capabilities to rapidly develop and field free-form, two-way, speech-to-speech translation systems that enable speakers of different languages to communicate with one another in real-world tactical situations without an interpreter. Evaluations of these technologies are a significant part of the program and DARPA has asked the National Institute of Standards and Technology (NIST) to lead this effort. This article presents the experimental design of the TRANSTAC evaluations and the metrics, both quantitative and qualitative, that were used to comprehensively assess the systems’ performance.

Reading Comprehension of Flemish Deaf Children in Belgium: Sources of Variability in Reading Comprehension after Cochlear Implantation

This paper reports the results of two studies of reading comprehension of Flemish children in Belgium. In the northern part of Belgium (Flanders), Dutch is the official language. The Dutch-speaking inhabitants of Flanders are called Flemish. Dutch is also the national language of the Netherlands. Despite both groups using Dutch, cultural differences between the groups are reflected in healthcare and education. Study 1 investigated the effect of these differences on the reading comprehension of deaf children. Reading levels of Flemish deaf children with cochlear implants (CI) (n = 30) or hearing aids (HA) (n = 44) were compared with reference data of Dutch deaf children with CI (n = 50) and HA (n = 500), and with a hearing norm group. Study 2 investigated sources of variability by examining the underlying processes of reading comprehension of good and poor Flemish deaf readers from the CI group in Study 1. Results of Study 1 showed significantly better reading levels for the two Flemish groups. These differences may reflect Belgian policy aiming at early implantation and the use of spoken language communication. The second study contrasted good and poor readers' working memory capacity scores, including verbal and non-verbal working memory scores, morphosyntactic ability, and phonological encoding in order to explore which underlying processes contributed to reading performance in the Flemish CI users. Results showed a tendency toward better morphosyntactic abilities and better working memory skills in the good readers. Three factors appear to explain the better reading of Flemish children with CIs: amount of spoken language input in communication, access to spoken language through a cochlear implant, and age at implantation.

Parental mode of communication is essential for speech and language outcomes in cochlear implanted children

Conclusion: The present study demonstrates a very strong effect of the parental communication mode on the auditory capabilities and speech/language outcome for cochlear implanted children. The children exposed to spoken language had higher odds of scoring high in all tests applied and the findings suggest a very clear benefit of spoken language communication with a cochlear implanted child. Objectives: The aim of the study was to identify factors associated with speech and language outcomes for cochlear implanted children and also to estimate the effect-related odds ratio for each factor in relation to the children's speech and language performances. Methods: Data relate to 155 prelingually deafened children with cochlear implant (CI). A test battery consisting of six different speech and language tests/assessments was used. Seven different factors were considered, i.e. hearing age, implantation age, gender, educational placement, ear of implantation, CI center, and communication mode. Logistic regression models and proportional odds models were used to analyze the relationship between the considered factors and test responses. Results: The communication mode at home proved essential to speech and language outcome, as children exposed to spoken language had markedly better odds of performing well in all tests, compared with children exposed to a mixture of spoken language and sign support, or sign language.

Introduction. The perception of speech: from sound to meaning

Spoken language communication is arguably the most important activity that distinguishes humans from non-human species. This paper provides an overview of the review papers that make up this theme issue on the processes underlying speech communication. The volume includes contributions from researchers who specialize in a wide range of topics within the general area of speech perception and language processing. It also includes contributions from key researchers in neuroanatomy and functional neuro-imaging, in an effort to cut across traditional disciplinary boundaries and foster cross-disciplinary interactions in this important and rapidly developing area of the biological and cognitive sciences.

Relationships Among Professionals' Knowledge, Experience, and Expectations Regarding Cochlear Implants

THE PRESENT STUDY examined the relationships between teachers' and communication clinicians' self-reported knowledge on cochlear implants and their expectations of CIs. The authors also explored these professionals' views regarding the child's communication mode, educational setting, and social options following cochlear implantation. The participants were 47 teachers of deaf students and 35 communication clinicians. The results showed that there were no significant differences between the two groups in self-reported knowledge on CI. Both groups knew very little about mapping, costs, or insurance, and reported good knowledge about candidacy. Expectations from CIs were moderate to high, and were significantly related to respondents' knowledge and specific experience with CIs. Most professionals in both groups supported spoken-language communication, individual inclusion, and social exposure to children with normal hearing as well as to children with hearing impairments.