Acoustic Cues Research Articles

Multimodal prosody: gestures and speech in the perception of prominence in Spanish

Multimodal communication cannot be properly understood without analyzing the natural interweaving of speech and gestures as it typically occurs in everyday spoken language, thus moving beyond studies that elicit gestures in the laboratory, most of which are also conducted for English. Therefore, this study addresses the effect of both visual and acoustic cues in the perception of prominence in Castilian Spanish using spontaneous speech from a TV talent-show. Four between-subjects experiments in each modality—audio-only and audiovisual—were conducted online, each including a different combination of manipulated cues: Exp1 (flat F0), Exp2 (flat intensity), and Exp3 (flat F0 + flat intensity), while all cues remained intact in the control experiment Exp0. Additionally, the capability of the different gesture phases to convey prominence was analyzed in their interaction with the acoustic cues. The results showed that, when prominence was perceived in manipulated stimuli, the effect of the visual information depended on the acoustic cues available in the signal and was also reduced when compared to non-manipulated stimuli, pointing to a strong integration of both modalities in prominence perception. In non-manipulated stimuli, all acoustic cues—except for spectral balance—played a role in the perception of prominence; however, when the visual information was added, it reduced the perceptual effect of the acoustic cues, and the main role played by duration was combined with that of the stroke phase of gestures.

Nasal/Oral Vowel Perception in French-Speaking Children With Cochlear Implants and Children With Typical Hearing.

The present study investigates the perception of vowel nasality in French-speaking children with cochlear implants (CIs; CI group) and children with typical hearing (TH; TH group) aged 4-12 years. By investigating the vocalic nasality feature in French, the study aims to document more broadly the effects of the acoustic limitations of CI in processing segments characterized by acoustic cues that require optimal spectral resolution. The impact of various factors related to children's characteristics, such as chronological/auditory age, age of implantation, and exposure to cued speech, has been studied on performance, and the acoustic characteristics of the stimuli in perceptual tasks have also been investigated. Identification and discrimination tasks involving French nasal and oral vowels were administered to two groups of children: 13 children with CIs (CI group) and 25 children with TH (TH group) divided into three age groups (4-6 years, 7-9 years, and 10-12 years). French nasal vowels were paired with their oral phonological counterpart (phonological pairing) as well as to the closest oral vowel in terms of phonetic proximity (phonetic pairing). Post hoc acoustic analyses of the stimuli were linked to the performance in perception. The results indicate an effect of the auditory status on the performance in the two tasks, with the CI group performing at a lower level than the TH group. However, the scores of the children in the CI group are well above chance level, exceeding 80%. The most common errors in identification were substitutions between nasal vowels and phonetically close oral vowels as well as confusions between the phoneme /u/ and other oral vowels. Phonetic pairs showed lower discrimination performance in the CI group with great variability in the results. Age effects were observed only in TH children for nasal vowel identification, whereas in children with CIs, a positive impact of cued speech practice and early implantation was found. Differential links between performance and acoustic characteristics were found within our groups, suggesting that in children with CIs, selective use of certain acoustic features, presumed to be better transmitted by the implant, leads to better perceptual performance. The study's results reveal specific challenges in children with CIs when processing segments characterized by fine spectral resolution cues. However, the CI children in our study appear to effectively compensate for these difficulties by utilizing various acoustic cues assumed to be well transmitted by the implant, such as cues related to the temporal resolution of stimuli. https://doi.org/10.23641/asha.25328704.

Spectrotemporal cues and attention jointly modulate fMRI network topology for sentence and melody perception

Speech and music are two fundamental modes of human communication. Lateralisation of key processes underlying their perception has been related both to the distinct sensitivity to low-level spectrotemporal acoustic features and to top-down attention. However, the interplay between bottom-up and top-down processes needs to be clarified. In the present study, we investigated the contribution of acoustics and attention to melodies or sentences to lateralisation in fMRI functional network topology. We used sung speech stimuli selectively filtered in temporal or spectral modulation domains with crossed and balanced verbal and melodic content. Perception of speech decreased with degradation of temporal information, whereas perception of melodies decreased with spectral degradation. Applying graph theoretical metrics on fMRI connectivity matrices, we found that local clustering, reflecting functional specialisation, linearly increased when spectral or temporal cues crucial for the task goal were incrementally degraded. These effects occurred in a bilateral fronto-temporo-parietal network for processing temporally degraded sentences and in right auditory regions for processing spectrally degraded melodies. In contrast, global topology remained stable across conditions. These findings suggest that lateralisation for speech and music partially depends on an interplay of acoustic cues and task goals under increased attentional demands.

Hierarchical graph contrastive learning of local and global presentation for multimodal sentiment analysis

Multi-modal sentiment analysis (MSA) aims to regress or classify the overall sentiment of utterances through acoustic, visual, and textual cues. However, most of the existing efforts have focused on developing the expressive ability of neural networks to learn the representation of multi-modal information within a single utterance, without considering the global co-occurrence characteristics of the dataset. To alleviate the above issue, in this paper, we propose a novel hierarchical graph contrastive learning framework for MSA, aiming to explore the local and global representations of a single utterance for multimodal sentiment extraction and the intricate relations between them. Specifically, regarding to each modality, we extract the discrete embedding representation of each modality, which includes the global co-occurrence features of each modality. Based on it, for each utterance, we build two graphs: local level graph and global level graph to account for the level-specific sentiment implications. Then, two graph contrastive learning strategies is adopted to explore the different potential presentations based on graph augmentations respectively. Furthermore, we design a cross-level comparative learning for learning local and global potential representations of complex relationships.

Auditory risk recognition is socially transmitted across territory borders in wild birds

Prey species commonly assess predation risk based on acoustic signals, such as predator vocalizations or heterospecific alarm calls. The resulting risk-sensitive decision-making affects not only the behavior and life-history of individual prey, but also has far-reaching ecological consequences for population, community, and ecosystem dynamics. Although auditory risk recognition is ubiquitous in animals, it remains unclear how individuals gain the ability to recognize specific sounds as cues of a threat. Here, it has been shown that free-living birds (Wood Warblers Phylloscopus sibilatrix) can learn to recognize unfamiliar, complex sounds (samples of punk rock songs) as cues of a threat from conspecifics holding adjacent territories during the spring breeding season. In a playback experiment, Wood Warblers initially ignored the unfamiliar sounds, but after repeatedly hearing that these sounds trigger alarm calling reaction of neighbors, most individuals showed an anti-predator response to them. Moreover, once learned soon after nestlings hatching, the anti-predator response of parents toward previously unfamiliar sounds was then retained over the entire nestlings rearing period. These results demonstrate that social learning via the association of unfamiliar sounds with known alarm signals enables the spread of anti-predator behavior across territory borders and provides a mechanism explaining the widespread abilities of animals to assess predation risk based on acoustic cues.

Developing an algorithm to characterize context-based speech patterns as cue production profiles

The surface articulation of an underlying phoneme commonly fluctuates depending on the context. For instance, the associated acoustic cues for a /t/ phoneme produced at the beginning of an utterance may be distinct from the acoustic cues of a /t/ phoneme that is preceded and followed by a vowel. Thus, for an individual speaker or speaker group, we can investigate what specific phonemic contexts result in the associated acoustic cues to be produced. An algorithm has been developed which matches a target phoneme to its corresponding acoustic cues. For a given database, we can recover all of the contexts in which a phoneme was produced and tabulate the various acoustic cue production patterns that arose. Finally, we label these patterns by their speech production type (e.g., standard production, flapping, etc.) and produce an output in the form of a cue production profile. As a result, this profile would allow for the association of a set of acoustic cues in specific phonemic contexts for an individual speaker or speaker group, such as speakers from a specific dialect region. The algorithm can also be expanded to include prosodic cues to account for how stress and intonation can affect phoneme production.

Acoustic cues in perception of reduced speech

In conversational speech, sounds undergo reduction and deletion, which challenges listeners to decode reduced forms using various cues. Previous research argues for the predominance of acoustic cues in a study of listeners’ perception of reduced speech such as “he’s” vs. “he was”, which can both be realized [ɨz]. Analyzing data from a previous perception experiment (Warner et al., Brain Sci., 2022), this study explores what types of acoustic cues listeners use to process reduced speech. Ambiguous [ɨz] with low second formant (F2) or long duration might be perceived as past due to an additional consonant (/w/) and potential contractions (low F2 as a cue to /w/). Low F2 is expected to result in smaller Bark F2-F1 and larger Bark F3-F2. Current study tests whether the stimuli show acoustic differences in the predicted direction and whether the formant and duration measures correlate with listeners’ identification of tense. It was found that there are no significant acoustic differences between past and present forms due to reduction, but that listeners significantly associated smaller Bark F2-F1 with past tense in singular verbs and longer duration with past tense in plural verbs. This confirms listeners' use of the predicted acoustic cues in perceiving reduced speech.

Automatic detection of nasal closure and nasal release landmark acoustic cues

This study describes the detection of nasal closure and nasal release landmarks, as part of a larger system for speech recognition based on acoustic cues. Landmarks are produced as a result of closures and releases in the oral region and are indicated by abrupt changes in the speech signal. Nasal closure and release landmarks have proven particularly challenging to detect and are the focus of this report. The process for implementing the nasal detection module includes extracting and processing a set of speech-related measurements, such as formant frequencies, spectral band energies, and their derivatives, from a large database of labeled speech files, and determining which of these measurements are potentially effective, using ANOVA analysis. Next, Gaussian mixture models are trained and tested on these measurements to classify nasal closures, nasal releases, and all other landmark cues. The resulting nasal closure and release landmark detection module will be used with other landmark modules for vowels, glides, fricative closures/releases, and stop closures/releases, as well as other acoustic cues to place and voicing, in the overall speech recognition system. The current performance of the module will be assessed and discussed.

Modeling syllable rhythms for interactive alignment studies

Many acoustic cues have been proposed as indicators and coordinators of upcoming turn-exchanges in conversational interactions, including syllable rhythm. Rhythmic coordination can help to explain close coordination of speech during these exchanges with intervals on the order of 250 ms or less, or with fluent and non-disruptive overlaps. While general questions surrounding speech rhythm remain unanswered, investigations of syllable alignment across partners in conversation can bypass such questions by assessing directly for patterns that align across vocal exchanges. We develop models of syllable rhythm for this purpose using hand-coded syllable durations from spontaneous conversations and from turn-taking responses elicited by recorded stimuli. Our approach builds on evidence that turn-takers begin planning their utterance approximately 0.5 s prior to turn endings. Syllable duration strings from appropriately identified frames are replaced by Gaussian curves to create oscillatory waveforms, and cross-correlation functions are applied to identify rhythmic continuities across exchanges. Patterns identified using this technique are explanatory of close coordination in the spontaneous exchanges, and supra-syllabic units such as the foot and phrase are identified as well. This syllable framework can be developed further by incorporating additional acoustic measures such as f;o and amplitude to build a multidimensional construct for interactional rhythm alignment.

Do individual differences correlate across speech perception tasks?

Speech perception requires listeners to take into account acoustic cues as well as lexical context and phonetic (coarticulatory) context. Individuals have been shown to vary in how they integrate these factors. To better understand the sources of these differences, we conducted three phoneme categorization tasks on speech continua with 82 native Canadian English speakers. Task 1 (lexical + coartic) embedded a /s-ʃ/ continuum in lexically biasing contexts (e.g., a(s)ume, a(ʃ)ure) followed by different coarticulatory contexts (rounded or unrounded vowels). Task 2 (lexical) had only lexical context cues for /ɛ/-/ɪ/ vowel continua (e.g., v(ɛ)st, k(ɪ)t). In task 3 (coartic), a /d/-/g/ stop continuum in nonsense syllables followed different coarticulatory contexts (/ar/ or /al/). We found those who used lexical context more used coarticulatory context less in task 1, consistent with prior research. However, this correlation disappears when examined across tasks 2 and 3. We also found no correlation between individual use of lexical and coarticulatory context across tasks, suggesting task dependency. Participants’ use of acoustic continua was positively correlated across tasks, indicating an individual trait for utilizing acoustic cues.

Perceptual weighting of acoustic cues to contrastive prosody for sentences in quiet and in noise

Prosody is used to mark important information in speech, yet despite this important role in everyday communication it is not a customary part of speech recognition testing in routine audiometry. Listeners might fail to perceive meaningful emphasis on a specific word signaled by prosodic cues despite repeating the words correctly. This study introduces a new paradigm for assessing perception of prosodic cues that are used to signify new/corrective information in a sentence. Stimuli consisted of spoken sentences where one word (in various sentence positions) was emphasized through acoustic modifications across the entire utterance, in a manner that corrected wrong information. Participants used a visual analog scale to mark the timing and degree of emphasis aligned with the target words. Perceptual data in this study are linked with acoustic measures of voice pitch contour, intensity, and duration to characterize how contrastive stress cues are recovered by listeners with and without cochlear implants, who are especially at risk for poorer pitch perception. Follow-up conditions used stimuli where acoustic cues (F0, duration, intensity) were manipulated independently to explore perceptual cue weighting, and the perseverance of these cues through background noise.

Automatic detection and labeling of glides for the English and Spanish language

This study outlines the development of a module for automatic detection of glide landmarks. We define glide landmarks as acoustic events that are observed during a narrowing in the oral cavity that does not result in cessation of airflow or in the conditions for turbulence noise. Glide landmarks are commonly associated with standard productions of underlying glides, a set that includes semivowels, liquids, and sounds produced by narrowing at the glottis, such as glottal stops, or the aspirant /h/. We lay out a framework that can be used to determine the acoustic measurements that are useful for detecting glide landmarks, and a Gausian Mixture Model is trained and tested, for automatic detection. A first investigation is carried out for /r/ production in English and Spanish. A closer look reveals that acoustic cues that describe abrupt onsets and offsets is useful for describing the trilled /r/ in Spanish, resulting in the need for an additional set of glide landmark cues for glide closure and glide release.

SoundGarden: Applying healthy soundscapes to support coral reef restoration

Coral reefs are a hub of ocean biodiversity supporting a range of socioeconomic and ecosystem services. Yet reefs and these important services are imperiled due to climate change and related stressors. Healthy coral reefs can be characterized by distinctive soundscapes, and there is increasing realization that acoustic cues are vital ecosystem components. Here we present a series of experiments examining if replaying healthy soundscapes can increase the settlement of coral larvae, a fundamental ecological process. Work was conducted in the U.S. Virgin Islands, leveraging a decade of soundscape studies on those reefs, and used a novel solar-powered acoustic playback system calibrated in sound pressure and particle motion. We studied larvae from three coral species, brooding Porites astreoides and Favia fragum, and broadcast-spawning Diploria labyrinthiformis. Larvae on degraded reefs enriched with healthy reef sounds settled at significantly higher rates compared to control sites that were not acoustically enriched. Acoustic enrichment settlement rates were influenced by received level, ecosystem properties and species biology reflecting the need for holistically evaluating the reef restoration process. Overall, this work outlines a new, potentially scalable means of supporting healthy reef habitat, and enhancing coral settlement on imperiled reefs undergoing restoration.

Spectral resolution and voice cue weighting in adults with normal hearing

Cochlear implant (CI) users use acoustic voice cues differently than normal hearing (NH) adults to identify a talker’s gender. Specifically, whereas NH listeners weight a talker’s fundamental frequency (f0) and resonance (operationalized as vocal tract length or VTL) equally, CI users rely almost exclusively on f0. CI users’ abnormal cue weighting may partially arise due to degraded auditory information delivered by the CI. We hypothesized that altering the amount of spectral information in the signal impacts voice cue weighting in NH listeners. Thirty NH adults performed a gender identification task. Auditory stimuli were monosyllabic words synthesized to have one of five f0 values and one of five VTL values. Synthesized voices were processed using a 16-, 8-, and 4-channel noise-band vocoder. Perceptual weights for each voice cue were estimated as the coefficients for f0 and VTL in regression models, with higher coefficients corresponding to stronger perceptual weightings. Listeners relied more on f0 in conditions with less spectral resolution than in conditions with greater spectral resolution. Cue weights for VTL did not change across conditions. Results suggest that removing frequency information from the auditory signal can modify the extent to which listeners use f0 to identify the gender of a talker.

Visual cues influence prosody perception among individuals with cochlear implants

Understanding a talker’s intention conveyed through prosody is essential for successful speech communication. Although visual speech cues are typically characterized by lip movements, there are also visual gestures including head tilts and eyebrow raises that signal prosody. Listeners with cochlear implants (CIs) may rely on these visual cues to solidify their prosody perception and guard against misinterpretations, particularly because of their poor perception of voice pitch. This study used audio-visual recordings of sentences in which a talker sometimes focused one word to signify a change in meaning. After characterizing acoustic and visual prosody cues in this stimulus set, we made prosodically mismatched variants of the stimuli by transplanting prosodically focused audio onto broad focus (i.e., prosodically unfocused) video, and vice versa. Visual influences on prosody perception were measured using acoustic analysis of participant reproductions in a vocal mimicry task. Results show reproduction of F0 contour—a key acoustic marker of prosodic focus—is influenced by perception of visual prosody cues. Specifically, visual cues indicating a focused word led to higher peak F0 in vocal reproductions of that word in the sentence. These results underscore the importance of multisensory information in supporting prosody perception for listeners with hearing loss.

Role of spatial positioning strategy for speech recognition in a simulated environment: Effects of age and hearing loss

In realistic listening environments, the physical position of the listener and target talker, relative to competing talkers, affects the signal-to-noise ratio and availability of spatial cues for release from masking. While the importance of signal-to-noise ratio and spatial cues for speech recognition has been demonstrated in the laboratory, a listener’s ability to leverage these acoustic cues by manipulating their position in an environment has received less attention. This study evaluated how changes in listener position within a simulated environment effect speech recognition for four listener groups that differed in age (younger or older) and hearing status (normal or impaired hearing). The simulated environment consisted of 8–10 speech maskers, individually rendered under headphones for specific locations within the space, with target and listener positions that varied across conditions. The intensity of the target varied throughout testing to estimate the psychometric function for keyword recognition of each listener in each position. Subjective reports of preferred listening position were collected before and after testing to determine the extent to which these listener groups strategically position themselves to maximize their speech recognition in noise. Results will be discussed in terms of the potential gains in speech recognition associated with optimal positioning strategies.

A novel method for measuring perception of suprasegmental cues in speech

While the perception of segmental cues in speech has been extensively studied, evidence on the perception of suprasegmental cues is largely missing. A critical barrier in this area is the challenge in measuring listeners’ ability to perceive a combination of multiple acoustic cues. The classic paradigm of discrimination is limited by its heavy reliance on auditory short-term memory and a long testing time. This study aims at examining the utility of a novel measure that is potentially more efficient and offers a more fine-grained account of perception of suprasegmental cues. Built on the evidence that listeners can shadow and rapidly imitate speech cues faithfully, we use the degree of alignment between listeners’ rapid imitation of suprasegmental cues and that of the speech stimuli as an index of perception quality. This alignment is mathematically quantified by the dynamic time warping (DTW) method, which is modified to simultaneously account for both the intensity and the fundamental frequency sequences to render more reliable alignment results. Data to date provided rich information regarding individual differences in perception, and results obtained from rapid imitation showed improved utility over those from the discrimination. The theoretical and clinical implication of this work is also discussed.

Particle motion polarization of offshore fish vocalizations, boat noise, and ambient soundscapes

Acoustic particle motion is an important acoustic cue for fish and aquatic invertebrate hearing. When reported for bioacoustics applications, it is often described as a magnitude only. However, particle motion is a vector quantity with polarization information (including directionality) relevant to detection and localization of sound cues by fishes and invertebrates. This study applied established metrics of particle motion polarization including ellipse amplitude, angle, orientation, and degree of polarization, to compare sounds of fish vocalizations with that of boat and ambient sounds at a 900 m-deep site (in the Atlantic Deepwater Ecosystem Observatory Network) offshore of Florida. Polarization metrics were computed for bivariate signals in the plane formed by the horizontal source-receiver and vertical axes. Boat sound polarization was quantified at multiple distances from the closest point of approach. Fish and boat sounds had narrower particle motion ellipses and larger amplitudes in the horizontal compared to ambient sounds that were more circular with larger relative vertical amplitudes. Fish and boat sounds were often strongly polarized, which may allow spatial release from masking given sufficient angular separation of sounds. This analysis framework has promising applications for monitoring in situ directionality of fish and other animal vocalizations and for modelling directional masking.

Where does the gamma go?: Acoustics of the non-labialized voiced velar approximant in Tlingit

The acoustic properties of the voiced velar approximant [ɰ] are poorly characterized, partly from understudy of relevant languages and partly from uncertainty about the sound itself. We investigate this sound in Tlingit, a critically endangered indigenous language of northwestern North America, where [ɰ] contrasts with labialized [w] and palatal [j] in some dialects but not in others where it was lost through sound change (reductive primary split with merge). We use audio recordings of spoken narratives spanning much of the 20th century from speakers with and without [ɰ] to characterize this sound both dialectally and diachronically. We measure formants (f1–f3) to identify place information, amplitude for oral aperture, and harmonic-to-noise ratio for manner. Existing descriptions and phonological patterns predict that [ɰ] is an approximant and not a fricative so we compare against known approximants for differences in turbulence and spectral properties. We also compare against labialized sounds including contrastive [w] for absence of lowered f3 to exclude lip rounding in [ɰ]. We argue that it is the intersection of these acoustic cues that distinguishes [ɰ] from similar sounds, but that these cues are unstable so that dialectal sound change arises from misperception in perturbations of their production.

Unexpected findings from L2 speech research can help us understand how humans process vowels

Research on nonnative speakers’ vowel production and perception has provided us with a number of surprising insights on how learners cope with nonnative vowels. Three of these initially unexpected, yet later solidly replicated, findings will be presented: (1) A decline in intelligibility of nonnative vowels as general proficiency improves (when an increase in intelligibility will be expected), (2) the use of acoustic cues in L2 vowel perception that cannot be attributed to transfer from the native language, and that are nonfunctional in the L2, and (3) the maintenance from infant speech perception in adult cross-language vowel perception of a bias favoring peripheral vowels. This presentation will discuss implications of these findings for our understanding of how vowel categories coexist in the minds of multilinguals, and of universally (native-language independent) preferred ways of vowel perception.