Acoustic Landmarks Research Articles

Tracking age-related changes in voice and speech production with Landmark-based analysis of speech.

Voice and speech production change with age, which can lead to potential communication challenges. This study explored the use of Landmark-based analysis of speech (LMBAS), a knowledge-based speech analysis algorithm based on Stevens' Landmark Theory, to describe age-related changes in adult speakers. The speech samples analyzed were sourced from the University of Florida Aging Voice Database, which included recordings of 16 sentences from the Speech Perception in Noise test of Bilger, Rzcezkowski, Nuetzel, and Rabinowitz [J. Acoust. Soc. Am. 65, S98-S98 (1979)] and Bilger, Nuetzel, Rabinowitz, and Rzeczkowski [J. Speech. Lang. Hear. Res. 27, 32-84 (1984)]. These sentences were read in quiet environments by 50 young, 50 middle-aged, and 50 older American English speakers, with an equal distribution of sexes. Acoustic landmarks, specifically, glottal, bursts, and syllabicity landmarks, were extracted using SpeechMark®, MATLAB Toolbox version 1.1.2. The results showed significant age effect on glottal and burst landmarks. Furthermore, the sex effect was significant for burst and syllabicity landmarks. While the results of LMBAS suggest its potential in detecting age-related changes in speech, increase in syllabicity landmarks with age was unexpected. This finding may suggest the need for further refinement and adjustment of this analytical approach.

A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online

Recent advances in automated face-tracking have made it possible for laboratory phonologists to measure lip motion using technology no more advanced than a common webcam. The present paper introduces a lip-tracking approach specifically designed for use in web-based research. The central innovation is a custom extension written for jsPsych, an existing JavaScript framework for running behavioral experiments online. This extension gathers data from the participant’s webcam and processes them through FaceMesh, an open-source, JavaScript face-tracker. Face-tracking happens on the fly inside the participant’s browser. Only time-stamped vertical or horizontal lip apertures are saved to the experiment server. That is, this extension allows experiments implemented in jsPsych to collect de-identified lip kinematics from participants seated at their own home computers. After explaining the core functionality of the extension, this paper presents two validation experiments. The first establishes that utterances with different lip kinematics result in very different lip aperture trajectories, and that the timing of a key kinematic landmark agrees with the timing of acoustic landmarks obtained for the same utterances. The second experiment replicates a vowel-priming result previously demonstrated using a prior face-tracking system and saved facial video. All associated scripts have been made publicly available.

Special anatomy series. Imaging inner ear structures with high-frequency ultrasound: Application to physical rehabilitation space medicine.

The objective of this paper is to document the feasibility of image acquisition, image optimization, and sonographic appearance of the exposed anatomic windows of cadaveric inner ear dissection for purposes of potential future clinical evaluation as part of the developing area of physical and rehabilitation space medicine. Cadaveric dissection of the inner ear was conducted with the goal of exposing areas relevant to vestibular balance. Middle and inner ear structures of 3 human cadavers were imaged with multiple broadband transducers, including emphasis with higher frequency transducers. The images were best optimized with 17 MHz and 22 MHz small footprint transducers. High-frequency ultrasound (US) images of the semicircular canals, vestibular and facial nerves, and utricles with reflected otoliths (otoconia) were obtained and reported in this article. Detailed visualization of both the vestibular nerve and facial nerve was accomplished, including identification of fascicular architecture. In addition, US reflection from the otoliths contained within the utricle was identified with sufficient clarity to provide surface measurements. Bony acoustic landmarks of the middle ear bones were identified by scanning externally from the tympanic membrane, including the dynamic movement of the bones with manual manipulation. US visualization has the potential to be an effective imaging modality to monitor potential changes to the otolith's size throughout extended space flight. To our knowledge, no prior study has reported US images of human inner ear structures.

An acoustic study of rhythmic synchronization with natural English speech

Sensorimotor synchronization as a means of studying rhythmic perception-action coupling has been extensively researched across a large number of temporally regular structures including music while little is known about synchronization with speech. The present study fills this gap by applying a sensorimotor synchronization paradigm to natural speech and studying acoustic landmarks that may serve as perceptual anchors of rhythmic movement in spoken sentences. Five rhythmically relevant types of acoustic landmarks were identified in twenty sentences of English containing syllables with vocalic and non-vocalic nuclei. The landmarks were either manually defined or algorithm-generated and included nucleus onsets, peaks and onsets of inter-syllabic and inter-stress timescales, moments of the fastest energy change (approximating the P-center location), and timepoints of combined pitch and periodic power. Sensorimotor synchronization data from 32 native English participants were examined with regards to the location of an increased synchronization activity in the proximity of the predefined landmarks. The results demonstrated that participants synchronized with syllable-size units regardless of the type of syllable nucleus (vowel or consonant) and that their taps were consistently timed close to nucleus onsets. Hereby, the manually defined nucleus onsets predicted synchronization peaks as well as the algorithm-generated moments of the fastest energy change around nucleus onsets (i.e., a model of the P-center location) did. In contrast, other landmarks did not constitute a stable acoustic anchor of sensorimotor synchronization with English speech. The synchronization performance was not influenced by either acoustic F0-information or by phonological tune specifications. These findings provide new evidence for the proposals that rhythmic attention in natural speech may be locked on to fast spectral changes within a syllable as the smallest structuring unit of prosodic hierarchy.

Phase Alignment of Low-Frequency Neural Activity to the Amplitude Envelope of Speech Reflects Evoked Responses to Acoustic Edges, Not Oscillatory Entrainment.

The amplitude envelope of speech is crucial for accurate comprehension. Considered a key stage in speech processing, the phase of neural activity in the theta-delta bands (1-10 Hz) tracks the phase of the speech amplitude envelope during listening. However, the mechanisms underlying this envelope representation have been heavily debated. A dominant model posits that envelope tracking reflects entrainment of endogenous low-frequency oscillations to the speech envelope. Alternatively, envelope tracking reflects a series of evoked responses to acoustic landmarks within the envelope. It has proven challenging to distinguish these two mechanisms. To address this, we recorded MEG while participants (n = 12, 6 female) listened to natural speech, and compared the neural phase patterns to the predictions of two computational models: an oscillatory entrainment model and a model of evoked responses to peaks in the rate of envelope change. Critically, we also presented speech at slowed rates, where the spectro-temporal predictions of the two models diverge. Our analyses revealed transient theta phase-locking in regular speech, as predicted by both models. However, for slow speech, we found transient theta and delta phase-locking, a pattern that was fully compatible with the evoked response model but could not be explained by the oscillatory entrainment model. Furthermore, encoding of acoustic edge magnitudes was invariant to contextual speech rate, demonstrating speech rate normalization of acoustic edge representations. Together, our results suggest that neural phase-locking to the speech envelope is more likely to reflect discrete representation of transient information rather than oscillatory entrainment.SIGNIFICANCE STATEMENT This study probes a highly debated topic in speech perception: the neural mechanisms underlying the cortical representation of the temporal envelope of speech. It is well established that the slow intensity profile of the speech signal, its envelope, elicits a robust brain response that "tracks" these envelope fluctuations. The oscillatory entrainment model posits that envelope tracking reflects phase alignment of endogenous neural oscillations. Here the authors provide evidence for a distinct mechanism. They show that neural speech envelope tracking arises from transient evoked neural responses to rapid increases in the speech envelope. Explicit computational modeling provides direct and compelling evidence that evoked responses are the primary mechanism underlying cortical speech envelope representations, with no evidence for oscillatory entrainment.

Persian Phone Recognition Using Acoustic Landmarks and Neural Network-based variability compensation methods

Persian Phone Recognition Using Acoustic Landmarks and Neural Network-based variability compensation methods

Improving passive acoustic target detection using machine learning classifiers

The ocean covers 70% of the Earth surface area, yet over 80% of it remains unexplored despite the advances in underwater acoustics and oceanography. Ocean exploration is critical for accurate climate model development, renewable energy applications, and in understanding the marine habitat. Further exploration necessitates improvements in underwater navigation with Autonomous Underwater Vehicles (AUVs). Utilizing acoustic landmarks can enhance AUV localization performance [Fula et al., Oceans IEEE/MTS (2018)]. Passive markers called Acoustic Identification (AID) Tags have unique and identifiable acoustic reflection signatures designed to function as landmarks [Satish et al., JASA 149 (2021)]. These targets can be detected by Match Filtering returns with template signals. Match Filtering performance is limited in the presence of strong interfering signals, and with changes in sound speeds that alters the temporal structure of these signatures. The application of a Machine Learning classifiers for detecting AID tag signatures can improve the localization performance. Through implementation of Logistic Regression, Deep Neural Networks, and Convolutional Neural Networks, the generalizability and superiority of these models is demonstrated. We report high accuracies (>90 %) in a multilabel (8 classes) classification task with signals with low SNR (–6 dB) and strong interference (+12 dB).

How much I moved: Robust biases in self-rotation perception

Vestibular cues are crucial to sense the linear and angular acceleration of our head in three-dimensional space. Previous literature showed that vestibular information precociously combines with other sensory modalities, such as proprioceptive and visual, to facilitate spatial navigation. Recent studies suggest that auditory cues may improve self-motion perception as well. The present study investigated the ability to estimate passive rotational displacements with and without virtual acoustic landmarks to determine how vestibular and auditory information interact in processing self-motion information. We performed two experiments. In both, healthy participants sat on a Rotational-Translational Chair. They experienced yaw rotations along the earth-vertical axis and performed a self-motion discrimination task. Their goal was to estimate both clockwise and counterclockwise rotations’ amplitude, with no visual information available, reporting whether they felt to be rotated more or less than 45°. According to the condition, vestibular-only or audio-vestibular information was present. Between the two experiments, we manipulated the procedure of presentation of the auditory cues (passive vs. active production of sounds). We computed the point of subjective equality (PSE) as a measure of accuracy and the just noticeable difference (JND) as the precision of the estimations for each condition and direction of rotations. Results in both experiments show a strong overestimation bias of the rotations, regardless of the condition, the direction, and the sound generation conditions. Similar to previously found heading biases, this bias in rotation estimation may facilitate the perception of substantial deviations from the most relevant directions in daily navigation activities.

Pushing the envelope: Evaluating speech rhythm with different envelope extraction techniques.

The amplitude of the speech signal varies over time, and the speech envelope is an attempt to characterise this variation in the form of an acoustic feature. Although tacitly assumed, the similarity between the speech envelope-derived time series and that of phonetic objects (e.g., vowels) remains empirically unestablished. The current paper, therefore, evaluates several speech envelope extraction techniques, such as the Hilbert transform, by comparing different acoustic landmarks (e.g., peaks in the speech envelope) with manual phonetic annotation in a naturalistic and diverse dataset. Joint speech tasks are also introduced to determine which acoustic landmarks are most closely coordinated when voices are aligned. Finally, the acoustic landmarks are evaluated as predictors for the temporal characterisation of speaking style using classification tasks. The landmark that performed most closely to annotated vowel onsets was peaks in the first derivative of a human audition-informed envelope, consistent with converging evidence from neural and behavioural data. However, differences also emerged based on language and speaking style. Overall, the results show that both the choice of speech envelope extraction technique and the form of speech under study affect how sensitive an engineered feature is at capturing aspects of speech rhythm, such as the timing of vowels.

Modified deep attractor neural networks for variability compensation in recognition tasks

Some brain functions can be described with attractor formation in the human neural system. Various biologically-inspired models have employed this characteristic for memory representation, pattern completion and, noise reduction. Attractor neural networks are amongst these models capable of settling to stable patterns called attractors using their recurrent connections. Deep attractor neural networks (DANets) are relatively new models that utilize their attractors to separate the sources in the mixtures. In this paper, a modified version of DANets, variability compensator attractor neural network (VCANets); is proposed. This model compensates variabilities which are a common source of performance degradation in recognition tasks. For this purpose, significant modifications were applied to the structure and training procedure of DANets, including attractors' calculation, multi-task learning, curriculum learning, mining of challenging samples, and special mini-batch training procedure. Experiments on the MNIST and MNIST-C datasets for handwritten digit recognition and FARSDAT dataset for acoustic landmark recognition show that VCANet can effectively improve recognition accuracy and reduce unwanted variabilities.

Binaural model analysis of wave-field-synthesis auralization over extended listening areas

This paper evaluates the performance of a recently proposed method to create auralizations with a 2D ray-tracing algorithm and publicly available floor plans. Using this method, acoustic landmarks were simulated for a 128-channel wave field synthesis (WFS) system. Late reverberation parameters are calculated using additional volumetric data. The approach allows the rapid sonic recreation of historical concert venues with adequate sound sources. The listeners can walk through these recreations over an extended user area (12 × 10 sqm), and the software suite can be used to calculate room acoustical parameters for various positions directly using a binaural rendering method or via the WFS simulation. To evaluate the performance of the auralization system binaural sound fields are rendered for various grid positions across the listening area and analyzed using a binaural model algorithm. The Binaurally Integrated Cross-correlation Auto-correlation Mechanism (BICAM) [J. Acoust. Soc. Am. 140(1), EL143] was selected for this purpose because it can estimate binaural activity patterns from a running signal with a 2-layer cross-correlation algorithm. Using this algorithm, the reflection patterns of early reflections and traditional room acoustical parameters (ISO 3382) were compared between the WFS auralization and direct binaural rendering method. [Work support by NSF IIS-1909229.]

Sonographic Location of Distal A1 Pulley by Means of a Bony Acoustic Landmark on the Proximal Phalanx: An Anatomic Study

To identify the anatomic relationship between the distal extent of the A1 pulley and the distalmost insertion point of the palmar plate into the base of the proximal phalanx at a metaphyseal bony ridge, the "P1 peak" (P1P), which is a bony acoustic landmark easily identifiable on sonography of the finger. We studied 48 fingers (no thumbs) from 12 fresh frozen cadavers with high-resolution ultrasound. A 20-gauge hypodermic needle was directed perpendicular to the long axis of the tendon sheath along a plumb line drawn at the level of the bony flare of the proximal metaphyseal-diaphyseal junction of the proximal phalanx, or the P1P, under high-resolution ultrasound guidance. Dissection was then performed, and the distance of the needle barrel and the actual distance of the P1P from the distal edge of the A1 pulley were recorded. The study demonstrated that accurate placement of the needle within 1 mm of the actual P1P was possible using ultrasound guidance and that the P1P was within 1 mm of the distal edge of the actual A1 pulley in all specimens studied. In all the specimens studied, we found that the P1P may be used as a clearly identifiable sonographic reference landmark to identify the distal extent of the A1 pulley. Using the P1P landmark as a guide to judge the completeness of ultrasound-guided percutaneous trigger finger release may reduce the risks of incomplete distal release and inadvertent release of the A2 pulley.

Speech-brain phase coupling is enhanced in low contextual semantic predictability conditions

Semantic prediction and cortical entrainment to the acoustic landmarks of the speech envelope are two fundamental yet qualitatively different mechanisms that facilitate speech comprehension. However, it is not clear how and to what extent those mechanisms interact with each other. On the one hand, richer semantic context could enhance the perceptual representation of a predictable stimulus, thus improving speech entrainment. On the other hand, pre-activating an upcoming item could inhibit further bottom-up analyses to minimize processing costs, thus weakening speech entrainment. To test these competing hypotheses, we recorded EEG activity from 27 participants while they listened to a 14-min recording of text reading. The passage contained target words presented twice: once in a highly constraining and once in a minimally constraining context. First, we measured event related potentials on target words in the two conditions. In line with previous research, we showed that semantic predictability modulated the N400 amplitude: words in minimally constraining contexts elicited larger negative amplitudes than words in highly constraining contexts between 250 and 450 ms. Second, we evaluated speech entrainment effects by analyzing phase alignment between neural activity and the envelope of target words. Importantly, we found increased speech entrainment for words in minimally constraining compared to highly constraining contexts between 400 and 450 ms. Both effects were located in central electrodes and were significantly correlated. Our results indicate a trade-off between semantic pre-activation and cortical entrainment to speech and support the cost minimization hypothesis.

Low-Frequency Neural Tracking of Natural Speech Envelope Reflects the Convolution of Evoked Responses to Acoustic Edges, Not Oscillatory Entrainment

The amplitude envelope of speech is crucial for accurate comprehension, and several studies have shown that the phase of neural activity in the theta-delta bands (1 - 10 Hz) tracks the phase of the speech amplitude envelope during listening, a process referred to as envelope tracking. However, the mechanisms underlying envelope tracking have been heavily debated. A dominant model posits that envelope tracking reflects continuous entrainment of endogenous low-frequency oscillations to the speech envelope. However, it has proven challenging to distinguish this from the alternative model that envelope tracking reflects a convolution of evoked responses to acoustic landmarks within the envelope. To address this, we recorded magnetoencephalography while participants listened to natural and slowed speech. First, we found that peaks in the rate of envelope change, acoustic edge landmarks, induced a strong evoked response. We then contrasted central diverging predictions of evoked response and oscillatory models regarding the spectral and temporal extent of neural phase locking in the theta-delta range at different speech rates. Our analyses revealed transient theta phase locking at both speech rates and delta phase locking in slow speech. This pattern was predicted by the evoked response model but not by oscillatory entrainment. Furthermore, the encoding of acoustic edge magnitudes was invariant to contextual speech rate, demonstrating speech rate normalization of acoustic edge representations. Taken together, our results show that an evoked response model provides a better account of neural phase locking to the speech envelope than oscillatory entrainment.

Linguistic Structure and Meaning Organize Neural Oscillations into a Content-Specific Hierarchy.

Neural oscillations track linguistic information during speech comprehension (Ding et al., 2016; Keitel et al., 2018), and are known to be modulated by acoustic landmarks and speech intelligibility (Doelling et al., 2014; Zoefel and VanRullen, 2015). However, studies investigating linguistic tracking have either relied on non-naturalistic isochronous stimuli or failed to fully control for prosody. Therefore, it is still unclear whether low-frequency activity tracks linguistic structure during natural speech, where linguistic structure does not follow such a palpable temporal pattern. Here, we measured electroencephalography (EEG) and manipulated the presence of semantic and syntactic information apart from the timescale of their occurrence, while carefully controlling for the acoustic-prosodic and lexical-semantic information in the signal. EEG was recorded while 29 adult native speakers (22 women, 7 men) listened to naturally spoken Dutch sentences, jabberwocky controls with morphemes and sentential prosody, word lists with lexical content but no phrase structure, and backward acoustically matched controls. Mutual information (MI) analysis revealed sensitivity to linguistic content: MI was highest for sentences at the phrasal (0.8-1.1 Hz) and lexical (1.9-2.8 Hz) timescales, suggesting that the delta-band is modulated by lexically driven combinatorial processing beyond prosody, and that linguistic content (i.e., structure and meaning) organizes neural oscillations beyond the timescale and rhythmicity of the stimulus. This pattern is consistent with neurophysiologically inspired models of language comprehension (Martin, 2016, 2020; Martin and Doumas, 2017) where oscillations encode endogenously generated linguistic content over and above exogenous or stimulus-driven timing and rhythm information.SIGNIFICANCE STATEMENT Biological systems like the brain encode their environment not only by reacting in a series of stimulus-driven responses, but by combining stimulus-driven information with endogenous, internally generated, inferential knowledge and meaning. Understanding language from speech is the human benchmark for this. Much research focuses on the purely stimulus-driven response, but here, we focus on the goal of language behavior: conveying structure and meaning. To that end, we use naturalistic stimuli that contrast acoustic-prosodic and lexical-semantic information to show that, during spoken language comprehension, oscillatory modulations reflect computations related to inferring structure and meaning from the acoustic signal. Our experiment provides the first evidence to date that compositional structure and meaning organize the oscillatory response, above and beyond prosodic and lexical controls.

Novel Virtual Reality System for Auditory Tasks in Head-fixed Mice.

An emerging corpus of research seeks to use virtual realities (VRs) to understand the neural mechanisms underlying spatial navigation and decision making in rodents. These studies have primarily used visual stimuli to represent the virtual world. However, auditory cues play an important role in navigation for animals, especially when the visual system cannot detect objects or predators. We have developed a virtual reality environment defined exclusively by free-field acoustic landmarks for head-fixed mice. We trained animals to run in a virtual environment with 3 acoustic landmarks. We present evidence that they can learn to navigate in our context: we observed anticipatory licking and modest anticipatory slowing preceding the reward region. Furthermore, we found that animals were highly aware of changes in landmark cues: licking behavior changed dramatically when the familiar virtual environment was switched to a novel one, and then rapidly reverted to normal when the familiar virtual environment was re-introduced, all within the same session. Finally, while animals executed the task, we performed in-vivo calcium imaging in the CA1 region of the hippocampus using a modified Miniscope.org system. Our experiments point to a future in which auditory virtual reality can be used to expand our understanding of the neural bases of audition in locomoting animals and the variety of sensory cues which anchor spatial representations in a new virtual environment.

A landmark-cue-based approach to analyzing the acoustic realizations of American English intervocalic flaps.

This study examines the acoustic realizations of American English intervocalic flaps in the TIMIT corpus, using the landmark-critical feature-cue-based framework. Three different acoustic patterns of flaps are described: (i) both closure and release landmarks present, (ii) only the closure landmark present, and (iii) both landmarks deleted. The patterns occur consistently across several phonological and morphological conditions but vary with sociolinguistic factors, including speaker dialect and gender. This method of analysing speech at the level of acoustic landmarks and other individual cues to distinctive features contributes to a deeper understanding of how speakers and listeners employ systematic variation in phonetic detail in speech processing.

Machine learning challenges in bat biosonar

Many bat species thrive in complex natural environments where their biosonar pulses trigger echoes with complex, unpredictable waveforms. In most cases, it remains unknown how the bats obtain sensory information they need from such clutter echoes. Machine learning methods that can extract relationships from large data sets could have a transformative impact on these research challenges. In particular, these methods hold considerable potential for answering the following questions: (i) What is the nature of biosonar echoes from complex environments; (ii) how can low-level navigation tasks such as contour-following and passageway finding be accomplished; (iii) what are acoustic landmarks for navigation and habitat selection; (iv) which clues exist for identifying prey in clutter; and (v) how does biosonar-based guidance in dense vegetation work at the system level? Machine learning methods are well suited for identifying models of echoes from complex vegetation that can be used to create large synthetic data sets with a known ground truth. Similarly, they can be employed to analyze (e.g., cluster) large physical echo datasets and to discover features associated with specific biosonar sensing tasks. Hence, machine learning could lead to the discovery of novel signal features to enable successful operation in much more complex environments.

A New Method of Simultaneous Localization and Mapping for Mobile Robots Using Acoustic Landmarks

The simultaneous localization and mapping (SLAM) problem for mobile robots has always been a hotspot in the field of robotics. Simultaneous localization and mapping for robots using visual sensors and laser radar is easily affected by the field of view and ground conditions. According to the problems of traditional sensors applied in SLAM, this paper presents a novel method to perform SLAM using acoustic signals. This method enables robots equipped with sound sources, moving within a working environment and interacting with microphones of interest, to locate itself and map the objects simultaneously. In our case, a method of microphone localization based on a sound source array is proposed, and it was applied as a pre-processing step to the SLAM procedure. A microphone capable of receiving sound signals can be directly used as a feature landmark of a robot observation model without feature extraction. Meanwhile, to eliminate the random error caused by hardware equipment, a sound settled in the middle of two microphones was applied as a calibration sound source to determine the value of the random error. Simulations and realistic experimental results demonstrate the feasibility and effectiveness of the proposed method.

Echolocating bats inspect and discriminate landmark features to guide navigation.

Landmark-guided navigation is a common behavioral strategy for way-finding, yet prior studies have not examined how animals collect sensory information to discriminate landmark features. We investigated this question in animals that rely on active sensing to guide navigation. Four echolocating bats (Eptesicus fuscus) were trained to use an acoustic landmark to find and navigate through a net opening for a food reward. In experimental trials, an object serving as a landmark was placed adjacent to a net opening and an object serving as a distractor was placed next to a barrier (covered opening). The location of the opening, barrier and objects were moved between trials, but the spatial relationships between the landmark and opening, and between the distractor and barrier were maintained. In probe trials, the landmark was placed next to a barrier, while the distractor was placed next to the opening, to test whether the bats relied on the landmark to guide navigation. Vocal and flight behaviors were recorded with an array of ultrasound microphones and high-speed infrared motion-capture cameras. All bats successfully learned to use the landmark to guide navigation through the net opening. Probe trials yielded an increase in both the time to complete the task and the number of net crashes, confirming that the bats relied largely on the landmark to find the net opening. Further, landmark acoustic distinctiveness influenced performance in probe trials and sonar inspection behaviors. Analyses of the animals' vocal behaviors also revealed differences between call features of bats inspecting landmarks compared with distractors, suggesting increased sonar attention to objects used to guide navigation.