Articulatory Movements Research Articles

Overt speech decoding from cortical activity: a comparison of different linear methods.

Speech BCIs aim at reconstructing speech in real time from ongoing cortical activity. Ideal BCIs would need to reconstruct speech audio signal frame by frame on a millisecond-timescale. Such approaches require fast computation. In this respect, linear decoder are good candidates and have been widely used in motor BCIs. Yet, they have been very seldomly studied for speech reconstruction, and never for reconstruction of articulatory movements from intracranial activity. Here, we compared vanilla linear regression, ridge-regularized linear regressions, and partial least squares regressions for offline decoding of overt speech from cortical activity. Two decoding paradigms were investigated: (1) direct decoding of acoustic vocoder features of speech, and (2) indirect decoding of vocoder features through an intermediate articulatory representation chained with a real-time-compatible DNN-based articulatory-to-acoustic synthesizer. Participant's articulatory trajectories were estimated from an electromagnetic-articulography dataset using dynamic time warping. The accuracy of the decoders was evaluated by computing correlations between original and reconstructed features. We found that similar performance was achieved by all linear methods well above chance levels, albeit without reaching intelligibility. Direct and indirect methods achieved comparable performance, with an advantage for direct decoding. Future work will address the development of an improved neural speech decoder compatible with fast frame-by-frame speech reconstruction from ongoing activity at a millisecond timescale.

Compensatory articulatory mechanisms preserve intelligibility in prodromal Parkinson's disease

IntroductionDysarthria is highly prevalent in patients with Parkinson's disease (PD) and speech changes have already been detected in patients with prodromal PD on the acoustic level. However, the present study directly tracks underlying articulatory movements with electromagnetic articulography to investigate early speech alterations on the kinematic level in isolated REM sleep behavior disorder (iRBD) and compares them to PD and control speakers. MethodsKinematic data of 23 control speakers, 22 speakers with iRBD, and 23 speakers with PD were collected. Amplitude, duration, and average speed of lower lip, tongue tip, and tongue body movements were analyzed. Naive listeners rated the intelligibility of all speakers. ResultsPatients with iRBD produced tongue tip and tongue body movements that were larger in amplitude and longer in duration compared to control speakers, while remaining intelligible. Compared to patients with iRBD, patients with PD had smaller, longer and slower tongue tip and lower lip movements, accompanied by lower intelligibility. Thus, the data indicate that the lingual system is already affected in prodromal PD. Furthermore, lower lip and especially tongue tip movements slow down and speech intelligibility decreases if motor impairment is more pronounced. ConclusionPatients with iRBD adjust articulatory patterns to counteract incipient motor detriment on speech to maintain their intelligibility level.

Corrective demonstrations and embodied resources for modeling speech sounds in aphasia speech-language therapy

This paper analyzes the way speech-language therapists instruct and correct aphasic speakers regarding the pronunciation of linguistic items and model pronunciation through auditory, verbal, visual, and tactile resources, thus configuring it as an embodied, multisensory experience. The data analysis shows that correction of a patient's productions is first initiated by therapists audibly, through repetition of the target item with specific prosodic features, such as loudness and length. When this type of cue proves to be ineffective, therapists perform instructions and corrective demonstrations through visual and haptic cues: By using their body as an “instructional tool,” they represent, with gestures and facial expressions, features of the target sound. They can also touch a patient's face to help them to correctly realize specific articulatory movements. This paper focuses on the hierarchy of embodied resources that are mobilized by therapists, in an ordered and incremental way, to model pronunciation. Moreover, it highlights the methods used to enhance patients' visual attention toward embodied resources and solve issues of bodily coordination. More broadly, this paper highlights the complex embodied nature of speaking and, by focusing on a pathological situation and a therapeutic setting, shows what participants treat as relevant for producing speech sounds in interaction.

Modeling trajectories of human speech articulators using general Tau theory

This paper presents an application of general Tau theory to the modeling and analysis of articulatory trajectories in speech. We evaluated the model using electromagnetic articulometry data from 12 native speakers of English reading a common text, where trajectories of the following sensors were fitted: lower and upper lips, jaw, and three tongue sensors. Additionally, we analyzed trajectories of the lip aperture signal. Our experiments show that the general Tau theory model gives a better fit than existing (i) methods based on critically damped oscillators, and (ii) a method based on sequential target approximation. These findings support the hypothesis of Tau-guided movements of articulators during speech production. In the second part of the paper, our Tau theory analysis shows that articulatory movements follow similar velocity profile distributions across speakers. In particular, the value of the shape parameter κ of the Tau theory equation is identically distributed across speakers, following a unimodal distribution. The statistical mode of the distribution corresponds to the value of κ that generates a symmetric velocity profile. The analysis of the statistical distribution of κ values also reveals that its variance decreases when greater articulatory effort is required, such that produced articulatory effort remains close to that predicted by the theoretical minimal cost function based on forces acting on the moving articulator. This provides new evidence that articulatory effort is optimized during speech production.

Synchronization of Surface Electromyography and 3D Electromagnetic Articulography Applied to Study the Biomechanics of the Mandible: Proof of Concept

Background. The aim of this study was to propose and establish the proof of concept of an approach to synchronize 3D Electromagnetic Articulography (3D-EMA) with Surface Electromyography (SEMG) based on the standard components of this equipment. Methods. The appropriate equipment and instruments were selected according to specifications stablished for this study. Once the necessary equipment was gathered, the proper conditions to synchronize the signals were created. Thus, we selected a SEMG with a switch module incorporated to be able to achieve synchronization of the signals. After the system setup was stablished, chewing tasks were recorded on a healthy volunteer, collecting a proof-of-concept database. The variability among recordings of the database were analyzed in terms of its standard deviation in order to detect possible interferences. Results. The analysis of the chewing task recordings obtained with the synchronized 3D EMA and SEMG signals in the present study did not reveal significant distortions, and all values were within those that had been given by the manufacturers of both of the systems. The method presented the advantage of using only components that are already included with the equipment employed. Conclusion. The method of analysis described in this paper is an effective tool that facilitates the investigation of mandibular movements synchronized in two domains: articulatory movements and electromyographic activity. Thus, it seems promising that it can be applied in different clinical situations to improve the analysis of the complexity of masticatory activity in addition to being able to generate new insights on this topic.

Different facial cues for different speech styles in Mandarin tone articulation

Visual facial information, particularly hyperarticulated lip movements in clear speech, has been shown to benefit segmental speech perception. Little research has focused on prosody, such as lexical tone, presumably because production of prosody primarily involves laryngeal activities not necessarily distinguishable through visible articulatory movements. However, there is evidence that head, eyebrow, and lip movements correlate with production of pitch-related variations. One subsequent question is whether such visual cues are linguistically meaningful. In this study, we compare movements of the head, eyebrows and lips associated with plain (conversational) vs. clear speech styles of Mandarin tone articulation to examine the extent to which clear-speech modifications involve signal-based overall exaggerated facial movements or code-based enhancement of linguistically relevant articulatory movements. Applying computer-vision techniques to recorded speech, visible movements of the frontal face were tracked and measured for 20 native Mandarin speakers speaking in two speech styles: plain and clear. Thirty-three head, eyebrow and lip movement features based on distance, time, and kinematics were extracted from each individual tone word. A random forest classifier was used to identify the important features that differentiate the two styles across tones and for each tone. Mixed-effects models were then performed to determine the features that were significantly different between the two styles. Overall, for all the four Mandarin tones, we found longer duration and greater movements of the head, eyebrows, and lips in clear speech than in plain speech. Additionally, across tones, the maximum movement happened relatively earlier in clear than plain speech. Although limited evidence of tone-specific modifications was also observed, the cues involved overlap with signal-based changes. These findings suggest that visual facial tonal modifications for clear speech primarily adopt signal-based general emphatic cues that strengthen signal saliency.

Psychological Peculiarities of Examination of Children with a Diagnosis of “Logoneurosis”, Complicated with Disorders of Sensory Sphere and Disorders of Perceptions

The purpose of the research is to show psychological peculiarities of exami-nation of children with a diagnosis of “logoneurosis”, complicated with disorders of sensory sphere and disorders of perceptions.research methods. The research methods were the method of observa-tion and the empirical study of patients. The place of organizing the empirical stage of our research was the psychiatric hospital 1 in Kyiv. With the help of clini-cal, pathopsychological and anamnestic methods 86 children in the age of 3­12 years old (the average age is 8 ± 0.5 years old) with logoneurosis in anamnesis were diagnosed. the results of the research. We proved, that adverse conditions, which fa-cilitate logoneurosis, are: physically weakened children; age features of brain activity; accelerated development of speech (3­4 years old), when the child’s communicative, cognitive and regulatory functions are developed rapidly un-der the influence of communication with adults; hidden mental imbalance of a child, increased reactivity as a result of not quite normal relationships with others; a conflict between the peculiarities of the environment and the degree of its awareness; lack of positive emotional contacts between adults and children; insufficient development of motility, sense of rhythm, facial expressions and ar-ticulatory movements; disorders of sensory sphere of the person and disorders of perceptions.conclusions. The ability of a child with logoneurosis to present speech free-ly depends not only on varying degrees of independence, but also on his/her readiness. So, the speech therapist should trace the features of the appearance of speech spasms depending on whether the child utters a complex or simple phrases, individual words or sounds. On the material of retelling the texts, it is recorded in which cases speech convulsions take a place: only at the beginning of the story, at the beginning of phrases, by use of individual words or sounds. It is being investigated whether speech convulsions do not depend on the level of speech volume. For this purpose, the examinee is asked to speak quietly, loudly, in a whisper.

Domain adaptation towards speaker-independent ultrasound tongue imaging based articulatory-to-acoustic conversion

In this paper, we endeavor to address an articulatory-to-acoustic issue which aims to estimate the mel-spectrogram of the acoustical signals, using midsagittal ultrasound tongue images of the vocal tract as input. Previous attempts employed statistical methods for the inversion between the articulatory movements and speech, while deep learning has begun to dominate this field. Despite the sustainable efforts that have been made, the mapping performance can be greatly varied for different speakers and most of the previous methods are constrained for the speaker-dependent scenario. Here, we present a novel approach towards speaker-independent mapping, which is inspired by the domain adaptation method. Specifically, we explore decoupling the spectrogram generation task and the speaker recognition task. Leveraging a novel designed loss function, we can improve the performance under the speaker-independent scenarios, through the adversarial learning strategy. To demonstrate the effectiveness of the proposed method, extensive experiments are conducted on the Tongue and Lips (TaL) corpus. Objective evaluation is conducted to compare the generated spectrograms and ground truth, using three evaluation metrics, including the MSE, SSIM, and CW-SSIM. The results indicate that our proposed method can achieve superior performance under the speaker-independent scenario, compared with competitive solutions. Our code is available at https://github.com/xianyi11/Articulatory-to-Acoustic-with-Domain-Adaptation.

Auditory feedback control of inter-articulator speech coordination: Evidence from tongue and jaw movements

In order to test whether and how immediate auditory feedback is involved in the coordinated action of sets of speech articulators, the current research quantified changes in the temporal and spatiotemporal relations between jaw and tongue tip movements in response to noise masking. Normal-hearing talkers recorded /tV#Cat/ utterances using electromagnetic articulography, with alternative V (/ɑ/-/ɛ) and C (/t/-/d/), across variation in production rate (fast-slow) and stress (first syllable stressed-unstressed). Approximately 240 utterances were produced in two conditions: normal listening and auditory feedback masking. Two kinematic measures were obtained: (1) timing of tongue-tip raising onset for medial C, relative to jaw opening-closing; and (2) angle of tongue-tip raising onset, relative to the jaw phase plane. In the normal listening condition, any manipulation that shortened the jaw opening-closing cycle reduced both the relative timing and phase angle of tongue-tip movement onset. In the masking condition, specific changes were observed in how consistently the phase angle of tongue-tip movement onset scaled with jaw opening–closing across rate and stress variation, but not in the relative timing of tongue-tip movement onset. Collectively, these findings suggest that the spatiotemporal phasing between articulator movements relies more on immediate auditory feedback than the relative timing between ongoing articulator movements.

Rapid movements at segment boundaries.

This paper reports on a one-to-one aspect of the articulatory-acoustic relationship, explaining how acoustic segment boundaries are a result of the rapid movements of the active articulators. In the acceleration profile, these are identified as acceleration peaks, which can be measured. To test the relationship, consonant and vowel segment durations are compared to articulatory posture intervals based on acceleration peaks, and time lags are measured on the alignment of the segment boundaries to the acceleration peaks. Strong relationships and short time lags are expected when the acceleration peaks belong to crucial articulators, whereas weak relationships are expected when the acceleration peaks belong to non-crucial articulators. The results show that lip posture intervals are indeed strongly correlated with [m], and tongue tip postures are strongly correlated with [n]. This is confirmed by the time lag results, which also reveal that the acoustic boundaries precede the acceleration peaks. Exceptions to the predictions are attributed to the speech material or the joint jaw-lip control unit. Moreover, the vowel segments are strongly correlated with the consonantal articulators while less correlated with the tongue body, suggesting that acceleration of crucial consonantal articulators determines not only consonant segment duration but also vowel segment duration.

Phonological and Articulatory Deficits in the Production of Novel Signs in Children With Developmental Language Disorder.

Sign language, like spoken language, incorporates phonological and articulatory (or motor) processing components. Thus, the learning of novel signs, like novel spoken word forms, may be problematic for children with developmental language disorder (DLD). In the present work, we hypothesize that phonological and articulatory deficits in novel sign repetition and learning would differentiate preschool-age children with DLD from their typical peers. Children with DLD (n = 34; aged 4-5 years) and their age-matched typical peers (n = 21) participated. Children were exposed to four novel signs, all iconic, but only two linked to a visual referent. Children imitatively produced these novel signs multiple times. We obtained measures of phonological accuracy and articulatory motion stability as well as of learning of the associated visual referent. Children with DLD showed an increased number of phonological feature (i.e., handshape, path, and orientation of the hands) errors when compared with their typical peers. While articulatory variability did not overall differentiate children with DLD from typical peers, children with DLD showed instability in one novel sign that obligated bimanual oppositional movement. Semantic aspects of novel sign learning were unaffected in children with DLD. Deficits that have been documented in phonological organization of spoken words in children with DLD are also evident in the manual domain. Analyses of hand motion variability suggest that children with DLD do not show a generalized motor deficit, but one that is restricted to the implementation of coordinated and sequential hand motion.

The Effects of Deep Brain Stimulation on Speech Motor Control in People With Parkinson's Disease.

Despite the overall benefits of deep brain stimulation (DBS) in Parkinson's disease (PD), its effects on speech production have been mixed when examined using auditory-perceptual and acoustic measures. This study investigated the effects of DBS on the lip and jaw kinematics during sentence production in individuals with dysarthria secondary to PD. Twenty-seven participants from three groups were included in the study: (a) individuals with PD and without DBS (PD group), (b) individuals with PD and with DBS (PD-DBS group), and (c) neurologically healthy control speakers (HC group). Lip and jaw movements during speech were recorded using optical motion capture and analyzed for path distance, speed, duration, articulatory stability, and interarticulator coordination. The PD-DBS group showed (a) increased path distance compared with the PD and HC groups and (b) increased speed compared with the PD group but not the HC group. Both PD and PD-DBS groups exhibited lengthened sentence duration compared with the HC group. Articulatory stability was greater for the two PD groups, PD and PD-DBS, compared with the HC group. Spatial, but not temporal, coordination was lower for the PD group than for the other two groups. The only kinematic changes between the DBS on and off conditions within the PD-DBS group were increases in spatial coordination. These data suggest that DBS primarily affects the amplitude scaling of articulatory movements, but not the temporal scaling, in individuals with PD. The findings are discussed with respect to the DBS-induced neural changes and their effects on speech motor control in PD.

Impacts of simplifying articulation movements imagery to speech imagery BCI performance

Objective. Speech imagery (SI) can be used as a reliable, natural, and user-friendly activation task for the development of brain-computer interface (BCI), which empowers individuals with severe disabilities to interact with their environment. The functional near-infrared spectroscopy (fNIRS) is advanced as one of the most suitable brain imaging methods for developing BCI systems owing to its advantages of being non-invasive, portable, insensitive to motion artifacts, and having relatively high spatial resolution. Approach. To improve the classification performance of SI BCI based on fNIRS, a novel paradigm was developed in this work by simplifying the articulation movements in SI to make the articulation movement differences clearer between different words imagery tasks. A SI BCI was proposed to directly answer questions by covertly rehearsing the word ‘是’ or ‘否’ (‘yes’ or ‘no’ in English), and an unconstrained rest task also was contained in this BCI. The articulation movements of SI were simplified by retaining only the movements of the jaw and lips of vowels in Chinese Pinyin for words ‘是’ and ‘否’. Main results. Compared with conventional speech imagery, simplifying the articulation movements in SI could generate more different brain activities among different tasks, which led to more differentiable temporal features and significantly higher classification performance. The average 3-class classification accuracies of the proposed paradigm across all 20 participants reached 69.6% and 60.2% which were about 10.8% and 5.6% significantly higher than those of the conventional SI paradigm operated in the 0–10 s and 0–2.5 s time windows, respectively. Significance. These results suggested that simplifying the articulation movements in SI is promising for improving the classification performance of intuitive BCIs based on speech imagery.

Decoding silent speech commands from articulatory movements through soft magnetic skin and machine learning.

Silent speech interfaces have been pursued to restore spoken communication for individuals with voice disorders and to facilitate intuitive communications when acoustic-based speech communication is unreliable, inappropriate, or undesired. However, the current methodology for silent speech faces several challenges, including bulkiness, obtrusiveness, low accuracy, limited portability, and susceptibility to interferences. In this work, we present a wireless, unobtrusive, and robust silent speech interface for tracking and decoding speech-relevant movements of the temporomandibular joint. Our solution employs a single soft magnetic skin placed behind the ear for wireless and socially acceptable silent speech recognition. The developed system alleviates several concerns associated with existing interfaces based on face-worn sensors, including a large number of sensors, highly visible interfaces on the face, and obtrusive interconnections between sensors and data acquisition components. With machine learning-based signal processing techniques, good speech recognition accuracy is achieved (93.2% accuracy for phonemes, and 87.3% for a list of words from the same viseme groups). Moreover, the reported silent speech interface demonstrates robustness against noises from both ambient environments and users' daily motions. Finally, its potential in assistive technology and human-machine interactions is illustrated through two demonstrations - silent speech enabled smartphone assistants and silent speech enabled drone control.

An Investigation into the Vowel Space Area and Vowel Intelligibility among English-speaking Children with Cochlear Implant in India

Abstract Background: Literature does show that studies of the vowel space area (VSA) can be helpful to know speech production skills in children with hearing impairment (HI) cochlear implant (CI). The need for the present study was to investigate speech production skills among children with CI, who receive Aural Rehab in nonnative language, i.e., English. The VSA offers an easy quick way to compare speech production accuracy among them to typically developing chidlren (TDC) peer group. Aim of the Study: The study aimed at investigating the VSA in F2 versus F1 space formed by vowels /a/, /i/, and /u/ in two groups of children (typically developing [TD] and children with CIs) and the intelligibility of the vowels spoken in the consonant-vowel-consonant (CVC) contexts. Methods: Two groups of subjects participated in the study, 100 children with CIs and 200 age-matched typically developing children. Recordings of vowels /a/, /i/, and /u/ in CVC were used for formant frequency extraction using Praat software, and the VSA was calculated. The recorded utterances were presented to three judges, for the estimation of the percentage of correct responses for identifying the vowel. The VSA for the vowels and the intelligibility scoring in percentage were valuated. Results: The results showed that the VSA among children with CI was significantly less compared to TDC. The reduced VSA represents limited articulatory movement in children with CI compared to children with TD for primary vowels embedded in CVC syllables. The vowel intelligibility for the three vowels in the CVC context was reduced for children with CI (70%) compared to that of children with TDC (95%). Discussion: Both first- and second-formant frequencies show distinct differences between both groups and consequently in the VSA. F1 is lower for /a/ for the CI group, while it is higher for vowel /i/ and /u/ for the CI group compared to the TDC group, while F2, the values were lower for /i/ but elevated significantly for both /a/ and /u/ compared to children in TDC. The intelligibility of their utterances (CVC) of children with CI was lower than that of TDC. Since the children were age- and gender-matched and were nonnative speakers of English, linguistic influences on measured values were minimum. Reasons for changed formant frequencies have to be explored further in terms of the influence of rehabilitation methods, visibility of articulatory movements required, and formant information coded by speech coding strategies. Correlation studies of the VSA and the speech intelligibility measures may assist in therapy outcomes and improve speech intelligibility in children with HI and modifications to mapping strategies employed. Conclusion: The F1 is lower for /a/ for the CI group, while it is higher for vowel /i/ and /u/ for the CI group compared to the TDC group, while F2, the values were lower for /i/ but elevated significantly for both /a/ and /u/ compared to children in TDC. The vowel triangle area was reduced in the speech of CI group subjects compared to that of the group of TDC.

Modified Eden-Lange Triple Tendon Transfer Procedure for Idiopathic Spinal Accessory Nerve Palsy and Scapular Winging

The scapula serves as the transition point at which the energy from the lower body and core transmits through the scapulothoracic articulation and the upper extremity. Spinal accessory nerve palsy leads to lateral winging of the scapula and abnormal scapulothoracic motion. Resultant abnormal motion and asynchrony of the scapulothoracic articulation can disrupt the upper extremity kinetic chain, leading to pain, atrophy, and dysfunction of the shoulder. Disruption of this kinetic chain can severely impair patients and athletes who engage in physically demanding activities. Conservative management alone with anti-inflammatories, lifestyle changes, and physical therapy has not produced encouraging results. Surgical management options include a tendon transfer (Eden-Lange, triple tendon transfer) or nerve repair. We present a case of an active duty service member with idiopathic spinal accessory nerve palsy treated with triple tendon transfer. After the triple tendon transfer, this patient experienced improvements in pain, range of motion, and function at 1-year postoperatively, also maintaining enough strength and function to pass their physical fitness assessments and remain on active duty. As such, these findings may suggest the triple tendon transfer procedure offers a viable option in the treatment of spinal accessory nerve palsy in the active patient populations who require a high level of physical function.

Simultaneous structures in sign languages: Acquisition and emergence.

The visual-gestural modality affords its users simultaneous movement of several independent articulators and thus lends itself to simultaneous encoding of information. Much research has focused on the fact that sign languages coordinate two manual articulators in addition to a range of non-manual articulators to present different types of linguistic information simultaneously, from phonological contrasts to inflection, spatial relations, and information structure. Children and adults acquiring a signed language arguably thus need to comprehend and produce simultaneous structures to a greater extent than individuals acquiring a spoken language. In this paper, we discuss the simultaneous encoding that is found in emerging and established sign languages; we also discuss places where sign languages are unexpectedly sequential. We explore potential constraints on simultaneity in cognition and motor coordination that might impact the acquisition and use of simultaneous structures.

Espaços potenciais-proximais virtuais: relato de experiência educacional em tempos de pandemia da Covid-19

It reports the activities developed, in a university extension program, to face the challenges of the physical distance imposed by the COVID-19 pandemic, with the creation of "Potential-proximate spaces". The objective report the authors' experiences in conducting the course " Potential-proximate spaces: reports of teaching experiences in Covid-19 pandemics times". The technological/virtual resources employed were available and used in the group's research, at undergraduate and graduate levels. The referred course sought to establish dialogues between the theoretical-methodological references of Winnicottian Psychoanalysis and Vygotskian Psychology which, despite their specificities, present themselves as exponents in the study of human development. The course was developed through synchronous and asynchronous activities, which culminated in the elaboration of interpretative syntheses, in conjunction with students and professors, with levels of complexity and deepening of the historical development of the conceptual propositions of the two authors studied. It resulted in the identification of approximations of potentiating relationships arising from interactions and experiences in the vygotskian Proximal/Imminent Development Zone and the winnicottian potencial spaces/environments; as well as conceptions of art and creativity as genuine, social and historical creative potentialities. In addition, the distance from epistemological bases with a more social or more individual aspect is highlighted and, finally, it suggests questions about the possibility of articulation between the Winnicottian conception of the innate vital and potential tendency to maturation, and the Vygotskian human potential with historical and social characteristics. The repercussions focus on the movement of articulation of different matrices of thought, moving away from the culture, in the field of Psychology, of distancing between different theoretical-methodological strands. Finally, it reflects on the planning of strategies to face and transpose the issues presented by the pandemic scenario to the maintenance of academic activities.

A segmentation-informed deep learning framework to register dynamic two-dimensional magnetic resonance images of the vocal tract during speech

ObjectiveDynamic magnetic resonance (MR) imaging enables visualisation of articulators during speech. There is growing interest in quantifying articulator motion in two-dimensional MR images of the vocal tract, to better understand speech production and potentially inform patient management decisions. Image registration is an established way to achieve this quantification. Recently, segmentation-informed deformable registration frameworks have been developed and have achieved state-of-the-art accuracy. This work aims to adapt such a framework and optimise it for estimating displacement fields between dynamic two-dimensional MR images of the vocal tract during speech. MethodsA deep-learning-based registration framework was developed and compared with current state-of-the-art registration methods and frameworks (two traditional methods and three deep-learning-based frameworks, two of which are segmentation informed). The accuracy of the methods and frameworks was evaluated using the Dice coefficient (DSC), average surface distance (ASD) and a metric based on velopharyngeal closure. The metric evaluated if the fields captured a clinically relevant and quantifiable aspect of articulator motion. ResultsThe segmentation-informed frameworks achieved higher DSCs and lower ASDs and captured more velopharyngeal closures than the traditional methods and the framework that was not segmentation informed. All segmentation-informed frameworks achieved similar DSCs and ASDs. However, the proposed framework captured the most velopharyngeal closures. ConclusionsA framework was successfully developed and found to more accurately estimate articulator motion than five current state-of-the-art methods and frameworks. SignificanceThe first deep-learning-based framework specifically for registering dynamic two-dimensional MR images of the vocal tract during speech has been developed and evaluated.

The timing of visual speech modulates auditory neural processing

In face-to-face communication, visual information from a speaker’s face and time-varying kinematics of articulatory movements have been shown to fine-tune auditory neural processing and improve speech recognition. To further determine whether the timing of visual gestures modulates auditory cortical processing, three sets of syllables only differing in the onset and duration of silent prephonatory movements, before the acoustic speech signal, were contrasted using EEG. Despite similar visual recognition rates, an increase in the amplitude of P2 auditory evoked responses was observed from the longest to the shortest movements. Taken together, these results clarify how audiovisual speech perception partly operates through visually-based predictions and related processing time, with acoustic–phonetic neural processing paralleling the timing of visual prephonatory gestures.