Acoustic Landmarks Research Articles

Acoustic landmarks contain more information about the phone string than other frames for automatic speech recognition with deep neural network acoustic model.

Most mainstream automatic speech recognition (ASR) systems consider all feature frames equally important. However, acoustic landmark theory is based on a contradictory idea that some frames are more important than others. Acoustic landmark theory exploits quantal nonlinearities in the articulatory-acoustic and acoustic-perceptual relations to define landmark times at which the speech spectrum abruptly changes or reaches an extremum; frames overlapping landmarks have been demonstrated to be sufficient for speech perception. In this work, experiments are conducted on the TIMIT corpus, with both Gaussian mixture model (GMM) and deep neural network (DNN)-based ASR systems, and it is found that frames containing landmarks are more informative for ASR than others. It is discovered that altering the level of emphasis on landmarks by re-weighting acoustic likelihood tends to reduce the phone error rate (PER). Furthermore, by leveraging the landmark as a heuristic, one of the hybrid DNN frame dropping strategies maintained a PER within 0.44% of optimal when scoring less than half (45.8% to be precise) of the frames. This hybrid strategy outperforms other non-heuristic-based methods and demonstrate the potential of landmarks for reducing computation.

Bats Use Path Integration Rather Than Acoustic Flow to Assess Flight Distance along Flyways

Navigation can be achieved using different strategies from simple beaconing to complex map-based movement [1-4]. Bats display remarkable navigation capabilities, ranging from nightly commutes of several kilometers and up to seasonal migrations over thousands of kilometers [5]. Many bats have been suggested to fly along fixed routes termed "flyways," when flying from their roost to their foraging sites [6]. Flyways commonly stretch along linear landscape elements such as tree lines, hedges, or rivers [7]. When flying along a flyway, bats must estimate the distance they have traveled in order to determine when to turn. This can be especially challenging when moving along a repetitive landscape. Some bats, like Kuhl's pipistrelles, which we studied here, have limited vision [8] and were suggested to rely on bio-sonar for navigation. These bats could therefore estimate distance using three main sensory-navigation strategies, all of which we have examined: acoustic flow, acoustic landmarks, or path integration. We trained bats to fly along a linear flyway and land on a platform. We then tested their behavior when the platform was removed under different manipulations, including changing the acoustic flow, moving the start point, and adding wind. We found that bats do not require acoustic flow, which was hypothesized to be important for their navigation [9-15], and that they can perform the task without landmarks. Our results suggest that Kuhl's pipistrelles use internal self-motion cues-also known as path integration-rather than external information to estimate flight distance for at least dozens of meters when navigating along linear flyways.

Early visual deprivation prompts the use of body-centered frames of reference for auditory localization

The effects of early visual deprivation on auditory spatial processing are controversial. Results from recent psychophysical studies show that people who were born blind have a spatial impairment in localizing sound sources within specific auditory settings, while previous psychophysical studies revealed enhanced auditory spatial abilities in early blind compared to sighted individuals. An explanation of why an auditory spatial deficit is sometimes observed within blind populations and its task-dependency remains to be clarified. We investigated auditory spatial perception in early blind adults and demonstrated that the deficit derives from blind individual’s reduced ability to remap sound locations using an external frame of reference. We found that performance in blind population was severely impaired when they were required to localize brief auditory stimuli with respect to external acoustic landmarks (external reference frame) or when they had to reproduce the spatial distance between two sounds. However, they performed similarly to sighted controls when had to localize sounds with respect to their own hand (body-centered reference frame), or to judge the distances of sounds from their finger. These results suggest that early visual deprivation and the lack of visual contextual cues during the critical period induce a preference for body-centered over external spatial auditory representations.

Beam Slice-Based Recognition Method for Acoustic Landmark With Multi-Beam Forward Looking Sonar

Autonomous underwater vehicles (AUVs) have been widely used for many risky underwater tasks. For these tasks, AUVs require navigational aids, such as active beacon-type underwater acoustic landmarks, which are the most widely used. It requires heavy installation loads and regular maintenance with high price sensors. In this paper, we propose a novel localization method in shallow water with a multi-beam forward looking sonar (MFLS), where positioning is based on passive-type acoustic landmarks. The proposed landmark comprises a combination of concrete pillars with a highly reliable and efficient recognition method for the real-time use of AUVs. The proposed landmark system has many advantages in the practical use, such as no maintenance, being strong on the marine bio-fouling, and light installation load with low price concrete pillars. The proposed recognition method is divided into two processes to verify the landmark. First, a pillarlike object is separated from the background, and the height of the pillar is calculated by the length of the shadow, which has a very high recognition rate and accuracy. Through modeling, the distance from the landmark to the MFLS can be calculated. The proposed method stochastically updates the navigation data based on the positional relationship with the landmark. The performance of the proposed landmark and its recognition method is verified through a water tank and field experiments using the AUV (Cyclops). The recognition rate and accuracy are also discussed.

Towards automated detection of similarities and differences in bilingual speakers

Bilingualism is increasingly the norm in the United States with at least 20% of Americans being bilingual. This change in linguistic demography has created a new challenge to accurately diagnose speech disorders among bilingual children. This study explores Acoustic Landmark Detection (ALD) system as an objective approach to characterizing similarities and differences in speech production in child speakers of Standard English and Jamaican Creole (JC). Eight JC-English bilingual children were recorded speaking eleven words three times in each language. Words were transcribed and entered into PROPH + to provide: (1) Phonological Mean Length of Utterance [pMLU], (2) phonotactic structure, and (3) Percent Consonants Correct (PCC). Landmarks were hand-marked to determine probable landmark sequences based on canonical word production. Canonical landmark sequences were aligned with detected landmark sequences in both languages using the Needleman-Wunsch global alignment algorithm. Analysis revealed that if PCC indicates JC and English words are different, the mismatch between landmark tends to be lower (higher production accuracy); if phonotactics indicate JC and English words are different, mismatch tends to be slightly higher (lower accuracy); and if pMLU indicates word differences, mismatch tends to be higher (lower accuracy). Our finding support traditional linguistic expectations regarding bilingual speakers’ similarities and differences.

Vowel recognition at fundamental frequencies up to 1 kHz reveals point vowels as acoustic landmarks.

The phonological function of vowels can be maintained at fundamental frequencies (fo) up to 880 Hz [Friedrichs, Maurer, and Dellwo (2015). J. Acoust. Soc. Am. 138, EL36-EL42]. Here, the influence of talker variability and multiple response options on vowel recognition at high fos is assessed. The stimuli (n = 264) consisted of eight isolated vowels (/i y e ø ε a o u/) produced by three female native German talkers at 11 fos within a range of 220-1046 Hz. In a closed-set identification task, 21 listeners were presented excised 700-ms vowel nuclei with quasi-flat fo contours and resonance trajectories. The results show that listeners can identify the point vowels /i a u/ at fos up to almost 1 kHz, with a significant decrease for the vowels /y ε/ and a drop to chance level for the vowels /e ø o/ toward the upper fos. Auditory excitation patterns reveal highly differentiable representations for /i a u/ that can be used as landmarks for vowel category perception at high fos. These results suggest that theories of vowel perception based on overall spectral shape will provide a fuller account of vowel perception than those based solely on formant frequency patterns.

Automated screening for speech disorders using acoustic landmark detection

Most children effortlessly learn how to coordinate movements for normal speech production. About one in twelve preschool-aged children, however, show delays in speech production capability that may put them at risk for academic and behavioral difficulties, if not identified and treated. Automated tools that can distinguish between children with and without speech and language impairments could serve as a useful clinical tool for early identification of speech related disorders in young children. In this study, we consider measures based on detecting sequences of acoustic landmarks characteristic of normal speech production over multisyllabic words and continuous speech samples. Ten normal adults, ten typical children and ten with speech disorders recorded twenty multisyllabic words and thirty-three sentences. Acoustic landmarks patterns within utterances and syllabic clusters are examined to characterize differences in landmark sequences between normal and disordered speech. Shannon’s Entropy and ROC analysis are used to evaluate the landmark patterns as potential diagnostic measures of a-typical speech production. We discuss these results and our future work toward developing a fully automated clinical screening tool.

Detecting glides and their place of articulation using speech-related measurements in a feature-cue-based model

An algorithm was developed for detecting glides (/w/, /j/, /r/, /l/, or /h/) in spoken English and detecting their place of articulation using an analysis of acoustic landmarks [Stevens 2002]. The system uses Gaussian mixture models (GMMs) trained on a subset of the TIMIT speech database annotated with acoustic landmarks. To characterize the glide tokens extracted from the speech samples, the following speech-related measurements were calculated: energy in four spectral bands (E1-E4), formant frequencies (F1-F4), and the time derivatives of E1-E4 (E1’-E4’); the fundamental frequency (F0) and magnitude difference of harmonics (H1-H2, H1-H4) were also included. GMMs were then trained on a subset of the tokens to learn the characteristics of each category for two distinct tasks: distinguishing glide landmarks from the set of all landmark types (identification task), and determining the place of articulation given a glide landmark (categorization task). The classifier used the maximum posterior probability of a speech sample conditioned on each of the trained GMMs. The performance of the algorithm was evaluated with median F-scores, and results suggest that the measurements at acoustic landmarks provide salient cues to glide detection and categorization.

How time-based alignment of realized acoustic landmarks and predicted landmarks improves analysis of feature cue modification patterns in speech

Acoustic landmarks are abrupt spectral changes that signal the underlying manner features of phonemes (Stevens 2002). Our goal in developing an automatic method to detect these landmarks is to create a robust, knowledge-based approach to phoneme extraction in automatic speech signal processing. One challenge in such an approach is posed by massive reductions, in which many landmarks and other feature cues are missing. Thus, there is a need to hand-label the acoustic landmarks that actually occur in the speech signal and align them with predicted landmarks. However, this often results in a discrepancy between the locations of labels for the automatically generated and hand-labeled landmarks, which leads to an inaccurate analysis of where realized landmarks occur with respect to word and phoneme interval boundaries. We attempted to solve this issue with a time-based alignment method derived from the minimum edit distance algorithm. The result improved alignment of the realized landmark labels with the predicted landmark labels, enabling a more accurate analysis of modifications in the hand-labeled (realized) landmark tiers.

Selecting frames for automatic speech recognition based on acoustic landmarks

Most mainstream Mel-frequency cepstral coefficient (MFCC) based Automatic Speech Recognition (ASR) systems consider all feature frames equally important. However, the acoustic landmark theory disagrees with this idea. Acoustic landmark theory exploits the quantal non-linear articulatory-acoustic relationships from human speech perception experiments and provides a theoretical basis of extracting acoustic features in the vicinity of landmark regions where an abrupt change occurs in the spectrum of speech signals. In this work, we conducted experiments, using the TIMIT corpus, on both GMM and DNN based ASR systems and found that frames containing landmarks are more informative than others during the recognition process. We proved that altering the level of emphasis on landmark and non-landmark frames, through re-weighting or removing frame acoustic likelihoods accordingly, can change the phone error rate (PER) of the ASR system in a way dramatically different from making similar changes to random frames. Furthermore, by leveraging the landmark as a heuristic, one of our hybrid DNN frame dropping strategies achieved a PER increment of 0.44% when only scoring less than half, 41.2% to be precise, of the frames. This hybrid strategy out-performs other non-heuristic-based methods and demonstrated the potential of landmarks for computational reduction for ASR.

A landmark-based approach to transcribing systematic variation in the implementation of /t, d/ flapping in American English

A model of human speech processing based on individual cues to distinctive features of phonemes, such as the acoustic landmarks (abrupt spectral changes) that signal manner features, is proposed to provide a more accurate account of American English flapping of /t/ and /d/ than an allophonic or phone-based model. To test this hypothesis, this study analyses the phonetic realization of /t, d/ in the context of flapping using the acoustic landmark cues of abrupt stop closure, abrupt stop release and glide-like amplitude minimum (Stevens 2002), in subsets of the TIMIT corpus. Results show that the majority of flapped variants of /t, d/ preserve their stop closure landmark and there are several cases where they preserve both of their stop landmarks (stop closure and stop release), while exhibiting the landmark modification for flapping (e.g., t-glide- + ). Additionally, flapped /t/ is more likely to maintain stop landmarks than flapped /d/. This is unexpected from the traditional view of the flap as a categorical phenomenon, and suggests that acoustic landmarks are useful in capturing systematic phonetic variation in flapping. It will be important to test whether this landmark-based analysis yields a better result in automatic speech recognition than (allo)phone-based approaches.

Toward an analysis of Spanish glides in the acoustic landmark framework

Stevens (2002) proposes that the distinctive feature [glide] is signaled by an acoustic landmark, i.e., an amplitude/F1 minimum, usually during a phonated region, but this hypothesis has not been tested extensively in languages other than American English. This study analyzes acoustic realizations of tapped /ɾ/ and trilled /r/ sounds in European Spanish, identifying a range of acoustic realizations for these consonants, including glides, and proposing criteria for identifying Spanish tap and trill landmarks in the speech signal. The speech sample of 200 tokens was drawn from the Albayzin corpus, which includes recordings of read Castillian Spanish from male and female speakers. Tokens were characterized by the number of amplitude minima (or occlusions) as well as the presence or absence of vocal fold vibration and noise. Additional factors analyzed include the rate of amplitude modification (tongue tip vibration), and contextual factors, including word and syllable position, stress, and consonant clusters. These moments of abrupt change (amplitude inflection points) provide cues to the manner features of the speaker’s intended words, and are hypothesized to play a significant role in perceptual processing and word recognition. These initial results for /r/ provide the basis for extension of this analysis to other Spanish glides.

Restoring an allocentric reference frame in blind individuals through echolocation

Recent psychophysical studies have described task-specific auditory spatial deficits in congenitally blind individuals. We investigated auditory spatial perception in congenitally blind children and adults during different auditory spatial tasks that required the localization of brief auditory stimuli with respect to either external acoustic landmarks (allocentric reference frame) or their own body (egocentric reference frame). Early blind participants successfully represented sound locations with respect to their body. However, they showed relative poor precision when compared to sighted participants during the localization of sound with respect to external auditory landmarks, suggesting that vision is crucial for an allocentric representation of the auditory space. In a separate study, we tested three congenitally blind individuals who used echolocation as a navigational strategy, to assess the benefit of echolocation on auditory spatial perception. Blind echolocators did not show the same impairment in auditory spatial localization reported for blind non-echolocators, but rather proved enhanced precision and accuracy as compared to blind non-echolocators and sighted participants. Our results suggest that echolocation can compensate for the spatial deficit reported in early blind individuals, likely by reactivating an allocentric reference frame needed to shape spatial representations similar to those generated by vision.

Automated anaysis of syllable complexity in children as an indicator of speech disorder

Speech disorders affecting intelligibility commonly occur in young children. One way to differentiate normal versus delayed speech development is to measure the ability to articulate increasingly complex syllables. We present a computer-assisted approach, Syllabic Cluster Analysis (SCA) as an objective measure of syllabic complexity. SCA uses clusters of acoustic landmarks to detect articulatory complexity in the production of syllables. Although most research using landmarks focuses on the lexical content of speech, SCA focuses on non-lexical differences which is well suited for analysis of speech with decreased intelligibility. Feasibility of this system to predict disordered speaker group is tested. Words recorded by normal adult (n = 10) and typical child (n = 20) speakers and children with speech disorders (n = 10) were ranked using a published word complexity measure to establish a high complexity word list (n= 20). Multinomial logistic regression models are fit for Landmarks per Syllabic Cluster (LM/SC) and Syllabic Cluster per Utterance (SC/UTTs) counts as speech complexity predictors. LM/SC counts are a significant predictor of disordered status in the model when measuring a short set of complex words (p <.001). Future work toward the development of automated diagnostic tools is discussed.

Acoustic cues to distinctive features are modified in the speech of typically-developing versus atypically developing children

Non-word repetition tasks have been used to diagnose children with various developmental difficulties with phonology, but these productions have not been phonetically analyzed to reveal the nature of the modifications produced by children diagnosed with SLI, autism spectrum disorder or dyslexia compared to those produced by typically-developing children. In this study, we compared the modification of predicted acoustic cues to distinctive features of manner, place and voicing for just under 30 children (ages 5-12), for the CN-Rep word inventory, in an extension of the earlier analysis in Levy et al. 2014. Feature cues, including abrupt acoustic landmarks (Stevens 2002) and other acoustic feature cues, were hand-labeled and analysis of factors that may influence feature cue modifications included position in the word, position in the syllable, word length measured in syllables, lexical stress, and manner type. Results suggest specific patterns of modification in specific contexts for specific clinical populations. These findings set the foundation for understanding how phonetic variation in speech arises in both typical and clinical populations, and for using this knowledge to develop tools to aid in more accurate and insightful diagnosis as well as improved intervention methods.

Feel the beat: Music exploits our brain's ability to predict and the dopamine-reward system to instill pleasure.

Music is universally cherished across humanity. Every culture has its songs and dances, everywhere in the world people listen to music; even small children clap their hands and move to the rhythm of the beat. Music moves us in every sense of the word. We just do not know why. The musical abilities of humankind have puzzled scientists for centuries, and Charles Darwin ranked them as “one of the most mysterious with which he is endowed”. Why do we make music, why do we like it, and how do we even perceive it? > The musical abilities of humankind have puzzled scientists for centuries, and Charles Darwin ranked them as “one of the most mysterious with which he is endowed”. “Music is art in time”, said Peter Vuust, a neuroscientist and music theoretician at Aarhus University in Denmark. Perceiving music has much to do with our sense of time and anticipation. Music is never perceived instantly: We need a few notes or beats before we detect a rhythm, and with the last few notes in mind, we anticipate what will happen next. Mostly, these anticipations will be met, but sometimes not. “Music plays with predictions, and this is what makes it so interesting to science”, explained Vuust, because it helps to investigate the brain's predictive timing mechanisms. “Predictive coding” is a conceptual framework of how the brain makes forecasts. It predicts incoming sensory information based on experience and context. This prediction is tightly coupled to the release of dopamine, the neurotransmitter of reward. In addition to predicting what will happen next, it is just as important to know exactly when it will happen, and an analogous system of “predictive timing” takes on this task. A school of fish or a flock of birds all turning at the same time, a dog …

Neural-inspired segmentation of audio streams into phone-like units

Recent work has shown that the human preauditory and/or auditory cortex is likely to play a role in acoustic landmark processing, such as the recognition of syllable and phoneme boundaries. Neurons appear to track acoustic envelopes with neural activity corresponding well with acoustic landmarks. These structures have also been observed in non-human primates, suggesting that acoustic landmark processing could be present in non-human primates and have an evolutionary role. Should landmark processing occur in non-humans, it could provide new methods for approaching animal communication. In this work, we use a simplified computer model to examine envelopes of acoustic signals that were filtered through a series of octave spaced band-pass filters. Peaks are detected in the signal envelopes with closely spaced peaks suppressed for a period of time after the first detection. Preliminary work shows an ability to detect human phoneme boundaries with a recall of 45% and a precision of 72% on the TIMIT speech corpus. The TIMIT corpus contains phoneme-level transcriptions and successful matches were defined as detecting landmarks within 20 ms of the over quarter million TIMIT phoneme boundaries. We also show qualitative results of the system for segmenting non-human primate calls into phone-like units.

Multimodal Integration of Spatial Information: The Influence of Object-Related Factors and Self-Reported Strategies.

Spatial representations are a result of multisensory information integration. More recent findings suggest that the multisensory information processing of a scene can be facilitated when paired with a semantically congruent auditory signal. This congruency effect was taken as evidence that audio-visual integration occurs for complex scenes. As navigation in our environment consists of a seamless integration of complex sceneries, a fundamental question arises: how is human landmark-based wayfinding affected by multimodality? In order to address this question, two experiments were conducted in a virtual environment. The first experiment compared wayfinding and landmark recognition performance in unimodal visual and acoustic landmarks. The second experiment focused on the congruency of multimodal landmark combinations and additionally assessed subject’s self-reported strategies (i.e., whether they focused on direction sequences or landmarks). We demonstrate (1) the equality of acoustic and visual landmarks and (2) the congruency effect for the recognition of landmarks. Additionally, the results point out that self-reported strategies play a role and are an under-investigated topic in human landmark-based wayfinding.

Using gated audiovisual speech perception to identify the temporal onset of coarticulation in production

The goal of the present work was to develop a sensitive, non-invasive method for accurately assessing the temporal scope of coarticulation in speech production. Building on work in audiovisual speech perception (Munhall and Tohkura, 1998; Moradi et al., 2013), we used a gating paradigm and human judges to identify the temporal onset of anticipatory lip rounding in simple SVO sentences produced by five adult females. The sentences were gated based on acoustic landmarks from the midpoint of the verb through to the midpoint of the object noun. Full sentences were also included. Judges were asked to decide whether the object noun rhymed with rounded “oop” or unrounded “ack.” Results indicated an earlier correct identification of rounding in the audiovisual condition compared to the control, audio-only condition. The audiovisual judgments also provided greater temporal specificity regarding the onset of coarticulation than could obtained with acoustic measurement or with a video-based kinematic measure. Insofa...

Blind individuals represent the auditory space in an egocentric rather than allocentric reference frame

It is not clear to date how visual deprivation affects auditory spatial perception. Recent psychophysical evidences described a spatial auditory deficit in congenitally blind individuals while some others found a spatial auditory improvement. Particularly, Gori, Sandini, Martinoli, & Burr (2014) and Vercillo, Milne, Gori, & Goodale (2015) reported that people who were born blind were less efficient in localizing sound sources with respect to auditory landmarks than sighted individuals. On the other side, blind people performed similarly or even better than sighted participants during the localization of single sound sources. We investigated auditory spatial perception in early blind using different auditory spatial tasks and found that blind individuals did not succeed in localizing sound sources in an external frame of reference. The performance of early blind was severely impaired during the localization of brief auditory stimuli with respect to acoustic landmarks (allocentric frame of reference) but was comparable to that one of sighted participants when they had to localize sounds with respect to their own body (egocentric reference frame). Our results suggest that, after early visual deprivation, auditory spatial perception is centered on an egocentric reference system.