Real-time Magnetic Resonance Imaging Data Research Articles

A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images

Real-time magnetic resonance imaging (RT-MRI) of human speech production is enabling significant advances in speech science, linguistics, bio-inspired speech technology development, and clinical applications. Easy access to RT-MRI is however limited, and comprehensive datasets with broad access are needed to catalyze research across numerous domains. The imaging of the rapidly moving articulators and dynamic airway shaping during speech demands high spatio-temporal resolution and robust reconstruction methods. Further, while reconstructed images have been published, to-date there is no open dataset providing raw multi-coil RT-MRI data from an optimized speech production experimental setup. Such datasets could enable new and improved methods for dynamic image reconstruction, artifact correction, feature extraction, and direct extraction of linguistically-relevant biomarkers. The present dataset offers a unique corpus of 2D sagittal-view RT-MRI videos along with synchronized audio for 75 participants performing linguistically motivated speech tasks, alongside the corresponding public domain raw RT-MRI data. The dataset also includes 3D volumetric vocal tract MRI during sustained speech sounds and high-resolution static anatomical T2-weighted upper airway MRI for each participant.

Complexity of vocal tract shaping in glossectomy patients and typical speakers: A principal component analysis.

The glossectomy procedure, involving surgical resection of cancerous lingual tissue, has long been observed to affect speech production. This study aims to quantitatively index and compare complexity of vocal tract shaping due to lingual movement in individuals who have undergone glossectomy and typical speakers using real-time magnetic resonance imaging data and Principal Component Analysis. The data reveal that (i) the type of glossectomy undergone largely predicts the patterns in vocal tract shaping observed, (ii) gross forward and backward motion of the tongue body accounts for more change in vocal tract shaping than do subtler movements of the tongue (e.g., tongue tip constrictions) in patient data, and (iii) fewer vocal tract shaping components are required to account for the patients' speech data than typical speech data, suggesting that the patient data at hand exhibit less complex vocal tract shaping in the midsagittal plane than do the data from the typical speakers observed.

A modular architecture for articulatory synthesis from gestural specification.

This paper proposes a modular architecture for articulatory synthesis from a gestural specification comprising relatively simple models for the vocal tract, the glottis, aero-acoustics, and articulatory control. The vocal tract module combines a midsagittal statistical analysis articulatory model, derived by factor analysis of air-tissue boundaries in real-time magnetic resonance imaging data, with an αβ model for converting midsagittal section to area function specifications. The aero-acoustics and glottis models were based on a software implementation of classic work by Maeda. The articulatory control module uses dynamical systems, which implement articulatory gestures, to animate the statistical articulatory model, inspired by the task dynamics model. Results on synthesizing vowel-consonant-vowel sequences with plosive consonants, using models that were built on data from, and simulate the behavior of, two different speakers are presented.

Speed-accuracy tradeoffs in human speech production.

Speech motor actions are performed quickly, while simultaneously maintaining a high degree of accuracy. Are speed and accuracy in conflict during speech production? Speed-accuracy tradeoffs have been shown in many domains of human motor action, but have not been directly examined in the domain of speech production. The present work seeks evidence for Fitts’ law, a rigorous formulation of this fundamental tradeoff, in speech articulation kinematics by analyzing USC-TIMIT, a real-time magnetic resonance imaging data set of speech production. A theoretical framework for considering Fitts’ law with respect to models of speech motor control is elucidated. Methodological challenges in seeking relationships consistent with Fitts’ law are addressed, including the operational definitions and measurement of key variables in real-time MRI data. Results suggest the presence of speed-accuracy tradeoffs for certain types of speech production actions, with wide variability across syllable position, and substantial variability also across subjects. Coda consonant targets immediately following the syllabic nucleus show the strongest evidence of this tradeoff, with correlations as high as 0.72 between speed and accuracy. A discussion is provided concerning the potentially limited applicability of Fitts’ law in the context of speech production, as well as the theoretical context for interpreting the results.

Movements of the Tongue during Lip Trills in Horn Players: Real-Time MRI Insights.

Movements inside the oral cavity during lip trilling in horn-playing are poorly understood and controversial, particularly with respect to pedagogy. Developments in real-time magnetic resonance imaging (RT-MRI) allow representations of oral cavity movement during lip trill performance on a MRI-compatible horn to be recorded and quantified. We present RT-MRI data on 11 highly skilled horn players obtained from serial images acquired at acquisition times of 33.3, 18.2, and 10.0 ms (i.e., at 30, 55 and 100 frames/sec) as they performed sixteenth note, whole-step trills between Eb4 and F4 (concert pitch) at two tempos, ~60 bpm and as fast as possible. For fast trilling (mean speed 178.3±24.7 bpm), 7 of 11 subjects exclusively utilized a tongue movement strategy, 3 used both a tongue and jaw strategy, and 1 exclusively used a jaw strategy. For trilling at ~60 bpm, all 11 subjects used a tongue movement strategy. We suggest using these movement strategies in teaching whole-step trills.

Movements of the Glottis During Horn Performance: A Pilot Study.

The functional role of the glottis in brass performance is poorly understood and controversial, particularly with respect to pedagogy. Technological limitations have prevented the non-invasive, systematic study of the glottis in the past, but developments in real-time magnetic resonance imaging (RT-MRI) allow representations of glottal movement during performance on a MRI-compatible horn to be recorded and quantified. We present RT-MRI data obtained on 6 advanced-level horn players from serial images acquired at an acquisition time of 33.3 ms as they performed sustained note exercises on three notes (concert Eb2, Eb4, and Bb4) at each of three dynamics (pp, mf, and ff) and a staccato exercise. An advanced-level trumpet player was also studied performing a modification of the staccato exercise designed to minimize vertical movement of the larynx. Glottal movements and positions in the coronal plane were analyzed using a customized MATLAB toolkit. In sustained note playing, there is a significant influence of dynamic on the degree of glottal adduction/abduction. There is greater adduction with softer notes, and greater abduction with louder notes. In slow staccato playing, glottal closure accompanies the cessation of each note and persists until iteration of the next note in the sequence. We demonstrate that RT-MRI provides a suitable method to identify and quantify glottal movement during horn playing. We further show that there is a direct relationship between dynamic level and glottal adduction/abduction, and that the glottis is involved in performing notes during slow staccato playing.

Bilabial Substitution Patterns during Consonant Production in a Case of Congenital Aglossia

Purpose: Congenital aglossia is a rare syndrome in which an individual is born without a tongue. The present paper examines articulatory details of the production of multiple consonants by an aglossic speaker. Method: Real-time magnetic resonance imaging data of the upper airway were collected from the aglossic speaker. Air-tissue boundaries were determined from the video sequences using a segmentation algorithm, and dynamics of vocal-tract constrictions and cross-dimensions were calculated. Results: The aglossic speaker produced the consonants /t, d, th, l ,r, f ,v, s, sh/ with a bilabial closure instead of a normal lingua-alveolar closure; however, in /t/and /d/ the overall vocal-tract configuration presented a cavity anterior to the constriction, which filtered transient and frication sources in a manner similar to normal alveolar production. Conclusion: The aglossic speaker, lacking a tongue apex, has developed a bilabial compensatory strategy to produce multiple consonants with her lips.

Sinogram-based coil selection for streak artifact reduction in undersampled radial real-time magnetic resonance imaging.

Streak artifacts are a common problem in radial magnetic resonance imaging (MRI). We therefore developed a method for automatically excluding receiver coil elements which lead to these artifacts. The proposed coil selection relates to real-time MRI data based on highly undersampled radial acquisitions. It exploits differences between high- and low-resolution sinograms reconstructed from datasets acquired during preparatory scans. Apart from phantom validations, the performance was assessed for real-time MRI studies of different human organ systems in vivo. The algorithm greatly reduces streak artifact strength without compromising image quality in other parts of the image. It is robust with respect to different experimental settings and fast to be included in the online reconstruction pipeline for real-time MRI. The proposed method enables a fast reduction of streak artifacts in radial real-time MRI.

On Short-Time Estimation of Vocal Tract Length from Formant Frequencies.

Vocal tract length is highly variable across speakers and determines many aspects of the acoustic speech signal, making it an essential parameter to consider for explaining behavioral variability. A method for accurate estimation of vocal tract length from formant frequencies would afford normalization of interspeaker variability and facilitate acoustic comparisons across speakers. A framework for considering estimation methods is developed from the basic principles of vocal tract acoustics, and an estimation method is proposed that follows naturally from this framework. The proposed method is evaluated using acoustic characteristics of simulated vocal tracts ranging from 14 to 19 cm in length, as well as real-time magnetic resonance imaging data with synchronous audio from five speakers whose vocal tracts range from 14.5 to 18.0 cm in length. Evaluations show improvements in accuracy over previously proposed methods, with 0.631 and 1.277 cm root mean square error on simulated and human speech data, respectively. Empirical results show that the effectiveness of the proposed method is based on emphasizing higher formant frequencies, which seem less affected by speech articulation. Theoretical predictions of formant sensitivity reinforce this empirical finding. Moreover, theoretical insights are explained regarding the reason for differences in formant sensitivity.

Detecting Nasal Vowels in Speech Interfaces Based on Surface Electromyography.

Nasality is a very important characteristic of several languages, European Portuguese being one of them. This paper addresses the challenge of nasality detection in surface electromyography (EMG) based speech interfaces. We explore the existence of useful information about the velum movement and also assess if muscles deeper down in the face and neck region can be measured using surface electrodes, and the best electrode location to do so. The procedure we adopted uses Real-Time Magnetic Resonance Imaging (RT-MRI), collected from a set of speakers, providing a method to interpret EMG data. By ensuring compatible data recording conditions, and proper time alignment between the EMG and the RT-MRI data, we are able to accurately estimate the time when the velum moves and the type of movement when a nasal vowel occurs. The combination of these two sources revealed interesting and distinct characteristics in the EMG signal when a nasal vowel is uttered, which motivated a classification experiment. Overall results of this experiment provide evidence that it is possible to detect velum movement using sensors positioned below the ear, between mastoid process and the mandible, in the upper neck region. In a frame-based classification scenario, error rates as low as 32.5% for all speakers and 23.4% for the best speaker have been achieved, for nasal vowel detection. This outcome stands as an encouraging result, fostering the grounds for deeper exploration of the proposed approach as a promising route to the development of an EMG-based speech interface for languages with strong nasal characteristics.

Real-time magnetic resonance imaging and electromagnetic articulography database for speech production research (TC).

USC-TIMIT is an extensive database of multimodal speech production data, developed to complement existing resources available to the speech research community and with the intention of being continuously refined and augmented. The database currently includes real-time magnetic resonance imaging data from five male and five female speakers of American English. Electromagnetic articulography data have also been presently collected from four of these speakers. The two modalities were recorded in two independent sessions while the subjects produced the same 460 sentence corpus used previously in the MOCHA-TIMIT database. In both cases the audio signal was recorded and synchronized with the articulatory data. The database and companion software are freely available to the research community.

Interspeaker Variability in Hard Palate Morphology and Vowel Production

Differences in vocal tract morphology have the potential to explain interspeaker variability in speech production. The potential acoustic impact of hard palate shape was examined in simulation, in addition to the interplay among morphology, articulation, and acoustics in real vowel production data. High-front vowel production from 5 speakers of American English was examined using midsagittal real-time magnetic resonance imaging data with synchronized audio. Relationships among hard palate morphology, tongue shaping, and formant frequencies were analyzed. Simulations were performed to determine the acoustical properties of vocal tracts whose area functions are altered according to prominent hard palate variations. Simulations revealed that altering the height and position of the palatal dome alters formant frequencies. Examinations of real speech data showed that palatal morphology is not significantly correlated with any formant frequency but is correlated with major aspects of lingual articulation. Certain differences in hard palate morphology can substantially affect vowel acoustics, but those effects are not noticeable in real speech. Speakers adapt their lingual articulation to accommodate palate shape differences with the potential to substantially affect formant frequencies, while ignoring palate shape differences with relatively little acoustic impact, lending support for acoustic goals of vowel production.

Aggregated motion estimation for real-time MRI reconstruction

In real-time MRI serial images are generally reconstructed from highly undersampled datasets as the iterative solutions of an inverse problem. While practical realizations based on regularized nonlinear inversion (NLINV) have hitherto been surprisingly successful, strong assumptions about the continuity of image features may affect the temporal fidelity of the estimated reconstructions. The proposed method for real-time image reconstruction integrates the deformations between nearby frames into the data consistency term of the inverse problem. The aggregated motion estimation (AME) is not required to be affine or rigid and does not need additional measurements. Moreover, it handles multi-channel MRI data by simultaneously determining the image and its coil sensitivity profiles in a nonlinear formulation which also adapts to non-Cartesian (e.g., radial) sampling schemes. The new method was evaluated for real-time MRI studies using highly undersampled radial gradient-echo sequences. AME reconstructions for a motion phantom with controlled speed as well as for measurements of human heart and tongue movements demonstrate improved temporal fidelity and reduced residual undersampling artifacts when compared with NLINV reconstructions without motion estimation. Nonlinear inverse reconstructions with aggregated motion estimation offer improved image quality and temporal acuity for visualizing rapid dynamic processes by real-time MRI.

Real‐time flow MRI of the aorta at a resolution of 40 msec

To evaluate a novel real-time phase-contrast magnetic resonance imaging (MRI) technique for the assessment of through-plane flow in the ascending aorta. Real-time MRI was based on a radial fast low-angle shot (FLASH) sequence with about 30-fold undersampling and image reconstruction by regularized nonlinear inversion. Phase-contrast maps were obtained from two (interleaved or sequential) acquisitions with and without a bipolar velocity-encoding gradient. Blood flow in the ascending aorta was studied in 10 healthy volunteers at 3 T by both real-time MRI (15 sec during free breathing) and electrocardiogram (ECG)-synchronized cine MRI (with and without breath holding). Flow velocities and stroke volumes were evaluated using standard postprocessing software. The total acquisition time for a pair of phase-contrast images was 40.0 msec (TR/TE = 2.86/1.93 msec, 10° flip angle, 7 spokes per image) for a nominal in-plane resolution of 1.3 mm and a section thickness of 6 mm. Quantitative evaluations of spatially averaged flow velocities and stroke volumes were comparable for real-time and cine methods when real-time MRI data were averaged across heartbeats. For individual heartbeats real-time phase-contrast MRI resulted in higher peak velocities for values above 120 cm s(-1). Real-time phase-contrast MRI of blood flow in the human aorta yields functional parameters for individual heartbeats. When averaged across heartbeats real-time flow velocities and stroke volumes are comparable to values obtained by conventional cine MRI.

An examination of the articulatory characteristics of prominence in function and content words using real-time magnetic resonance imaging

We examine the functional coupling between articulatory characteristics of prominence, such as articulator speed, and its acoustic characteristics, such as F0 and acoustic energy, for content words (nouns) and function words (e.g., articles, prepositions and conjunctions) using real-time magnetic resonance imaging data. We use Granger causality ideas to test the degree and direction of causal influence between the chosen articulatory and acoustic measures for function and content words. We further apply functional canonical correlation analysis to these measures to understand the covariant behavioral modes of the articulatory and acoustic measures. After controlling for word duration, we observe that articulatory speed generally has a significant causal influence on F0, especially for longer content words, but observe no such effect in the opposite direction. Notably, we do not observe this effect for function words in most cases. We further observe a tighter coupling of canonical weight functions of articulatory speed and acoustic-prominence characteristics (F0 and energy) for the content words as compared to the function words considered. These observations provide support for the hypothesis that prominence realized during content words may result from a close coupling between articulatory and acoustic characteristics such as articulatory speed and F0, with suggestions of a directional causal relationship. [Work supported by NIH.]

Co-registration of articulographic and real-time magnetic resonance imaging data for multimodal analysis of rapid speech

We propose a method for co-registrating speech articulatory/acoustic data from two modalities that provide complementary advantages. Electromagnetic Articulography (EMA) provides high temporal resolution (100 samples/second in WAVE system) and flesh-point tracking, while real-time Magnetic Resonance Imaging, rtMRI, (23 frames/second) offers a complete midsagittal view of the vocal tract, including articulated structures and the articulatory environment. Co-registration was achieved through iterative alignment in the acoustic and articulatory domains. Acoustic signals were aligned temporally using Dynamic Time Warping, while articulatory signals were aligned variously by minimization of mean total error between articulatometry data and estimated corresponding flesh points and by using mutual information derived from articulatory parameters for each sentence. We demonstrate our method on a subset of the TIMIT corpus elicited from a male and a female speaker of American English, and illustrate the benefits of co-registered multi-modal data in the study of liquid and fricative consonant production in rapid speech. [Supported by NIH and NSF grants.]

Statistical analysis of constriction task and articulatory posture variables during speech and pausing intervals using real-time magnetic resonance imaging

We have shown previously using real-time magnetic resonance imaging data (Ramanarayanan etal., 2011, ISSP Montreal) that it is more likely that articulatory posture variables (such as jaw angle) are controlled to achieve articulatory settings during pausing intervals in read speech than constriction task variables (such as lip aperture, tongue tip constriction degree, etc.). In this study, we extend this work to examine correlations between constriction task variables and articulatory posture variables during both speech and pause intervals. This work serves to deepen our understanding of the differences in postural motor control of the vocal tract observed during speaking and pausing. [Work supported by NIH.]

Rapid full volume data acquisition by real-time 3-dimensional echocardiography for assessment of left ventricular indexes in children: A validation study compared with magnetic resonance imaging

We sought to assess the feasibility, accuracy, and reproducibility of a rapid full volume acquisition strategy using real-time (RT) 3-dimensional (3D) echocardiography (3DE) for measurement of left ventricular (LV) volumes, mass, stroke volume (SV), and ejection fraction (EF) in children. A total of 19 healthy children (mean 10.6 +/- 2.8 years, 11 male and 9 female) were prospectively enrolled in this study. RT 3DE was performed using an ultrasound system to acquire full volume 3D dataset from the apical window with electrocardiographic triggering in 8 s/dataset. The images were processed offline using software. The LV endocardial and epicardial borders were traced manually to derive LV end-systolic volume, end-diastolic volume, mass, SV, and EF. Magnetic resonance imaging (MRI) studies were performed on a 1.5-T scanner using a breath hold 2-dimensional cine-FIESTA (fast imaging employing steady-state acquisition) sequence. All RT 3DE and MRI data were acquired successfully for analysis. Measurements of LV end-systolic volume, end-diastolic volume, mass, SV, and EF by RT 3DE correlated well by Pearson regression ( r = 0.86-0.97, P < .001) and agreed well by Bland-Altman analysis with MRI. The interobserver and intraobserver variability of RT 3DE measurements were less than 5%. This prospective study demonstrated that RT 3DE measurements of LV end-systolic volume, end-diastolic volume, mass, SV, and EF in children using rapid full volume acquisition strategy are feasible, accurate, and reproducible and are comparable with MRI measurements.