Non-speech Sounds Research Articles

Neuroimaging Findings for the Overnight Consolidation of Learned Non-native Speech Sounds

Abstract Research over the past two decades has documented the importance of sleep to language learning. Sleep has been suggested to play a role in establishing new speech representations as well; however, the neural mechanisms corresponding to sleep-mediated effects on speech perception behavior are unknown. In this study, we trained monolingual English-speaking adults to perceive differences between the Hindi dental vs. retroflex speech contrast in the evening. We examined the blood oxygen level dependent signal using functional magnetic resonance imaging during perceptual tasks on both the trained talker and on an untrained talker shortly after training, and again the next morning. We also employed diffusion tensor imaging to determine if individual differences in white matter structure could predict variability in overnight consolidation. We found greater activity in cortical regions associated with language processing (e.g., left insula) on the second day. Fractional anisotropy values in the anterior thalamic radiation and the uncinate fasciculus was associated with the magnitude of overnight change in perceptual behavior on the generalization (untrained) talker, after controlling for differences in sleep duration and initial learning. Our findings suggest that speech-perceptual information is subject to an overnight transfer of information to the cortex. Moreover, neural structure appears to be linked to individual differences in efficiency of overnight consolidation.

Utilization of Detection of Non-Speech Sound for Sustainable Quality of Life for Deaf and Hearing-Impaired People: A Systematic Literature Review

Several assistive technologies (ATs) have been manufactured and tested to alleviate the challenges of deaf or hearing-impaired people (DHI). One such technology is sound detection, which has the potential to enhance the experiences of DHI individuals and provide them with new opportunities. However, there is a lack of sufficient research on using sound detection as an assistive technology, specifically for DHI individuals. This systematic literature review (SLR) aims to shed light on the application of non-verbal sound detection technology in skill development for DHI individuals. This SLR encompassed recent, high-quality studies from the prestigious databases of IEEE, ScienceDirect, Scopus, and Web of Science from 2014 to 2023. Twenty-six articles that met the eligibility criteria were carefully analyzed and synthesized. The findings of this study underscore the significance of utilizing sound detection technology to aid DHI individuals in achieving independence, access to information, and safety. It is recommended that additional studies be conducted to explore the use of sound detection tools as assistive technology, to enhance DHI individual’s sustainable quality of life.

Training Child Learners on Nonnative Vowel Contrasts With Phonetic Training: The Role of Task and Variability

AbstractSubstantial research suggests that high variability (multitalker) phonetic training helps second language (L2) adults improve differentiation of challenging nonnative speech sounds. Is such training also useful for L2 children? Existing studies have mixed findings and important limitations. We investigate the potential benefits of computerized phonetic training for 50 Dutch 7‐year‐olds and 39 11‐year‐olds trained on English vowel contrasts in a 2‐week study in a classroom setting. Half received multitalker, and half received single‐talker input (i.e., high variability vs. low variability; HV vs. LV), with learning evaluated by a battery of tests. Both groups improved in training; however, 11‐year‐olds improved more. Moreover, 11‐year‐olds showed generalization to novel talkers, and 7‐year‐olds did not, with Bayes factor analyses providing evidence for the null. Generalisation in 11‐year‐olds was no greater following HV than LV input, with evidence for the null on one of two tasks where generalization was found. Results are discussed in terms of the interplay between age, task demands, and talker variability.

Individual differences in the consistency of neural and behavioural responses to speech sounds

There are documented individual differences among adults in the consistency of speech sound processing, both at neural and behavioural levels. Some adults show more consistent neural responses to speech sounds than others, as measured by an event-related potential called the frequency-following response (FFR); similarly, some adults show more consistent behavioural responses to native speech sounds than others, as measured by two-alternative forced choice (2AFC) and visual analog scaling (VAS) tasks. Adults also differ in how successfully they can perceive non-native speech sounds. Interestingly, it remains unclear whether these differences are related within individuals. In the current study, native English-speaking adults completed native phonetic perception tasks (2AFC and VAS), a non-native (German) phonetic perception task, and an FFR recording session. From these tasks, we derived measures of the consistency of participants’ neural and behavioural responses to native speech as well as their non-native perception ability. We then examined the relationships among individual differences in these measures. Analysis of the behavioural measures revealed that more consistent responses to native sounds predicted more successful perception of unfamiliar German sounds. Analysis of neural and behavioural data did not reveal clear relationships between FFR consistency and our phonetic perception measures. This multimodal work furthers our understanding of individual differences in speech processing among adults, and may eventually lead to individualized approaches for enhancing non-native language acquisition in adulthood.

Effect of Sound Genre on Emotional Responses for Adults With and Without Hearing Loss.

Adults with permanent hearing loss exhibit a reduced range of valence ratings in response to nonspeech sounds; however, the degree to which sound genre might affect such ratings is unclear. The purpose of this study was to determine if ratings of valence covary with sound genre (e.g., social communication, technology, music), or only expected valence (pleasant, neutral, unpleasant). As part of larger study protocols, participants rated valence and arousal in response to nonspeech sounds. For this study, data were reanalyzed by assigning sounds to unidimensional genres and evaluating relationships between hearing loss, age, and gender and ratings of valence. In total, results from 120 adults with normal hearing (M = 46.3 years, SD = 17.7, 33 males and 87 females) and 74 adults with hearing loss (M = 66.1 years, SD = 6.1, 46 males and 28 females) were included. Principal component analysis confirmed valence ratings loaded onto eight unidimensional factors: positive and negative social communication, positive and negative technology, music, animal, activities, and human body noises. Regression analysis revealed listeners with hearing loss rated some genres as less extreme (less pleasant/less unpleasant) than peers with better hearing, with the relationship between hearing loss and valence ratings being similar across genres within an expected valence category. In terms of demographic factors, female gender was associated with less pleasant ratings of negative social communication, positive and negative technology, activities, and human body noises, while increasing age was related to a subtle rise in valence ratings across all genres. Taken together, these results confirm and extend previous findings that hearing loss is related to a reduced range of valence ratings and suggest that this effect is mediated by expected sound valence, rather than sound genre.

Exploring the effectiveness of reward-based learning strategies for second-language speech sounds.

Adults struggle to learn non-native speech categories in many experimental settings (Goto, Neuropsychologia, 9(3), 317-323 1971), but learn efficiently in a video game paradigm where non-native speech sounds have functional significance (Lim & Holt, Cognitive Science, 35(7), 1390-1405 2011). Behavioral and neural evidence from this and other paradigms point toward the involvement of reinforcement learning mechanisms in speech category learning (Harmon, Idemaru, & Kapatsinski, Cognition, 189, 76-88 2019; Lim, Fiez, & Holt, Proceedings of the National Academy of Sciences, 116, 201811992 2019). We formalize this hypothesis computationally and implement a deep reinforcement learning network to map between environmental input and actions. Comparing to a supervised model of learning, we show that the reinforcement network closely matches aspects of human behavior in two experiments - learning of synthesized auditory noise tokens and improvement in speech sound discrimination. Both models perform comparably and the similarity in the output of each model leads us to believe that there is little inherent computational benefit to a reward-based learning mechanism. We suggest that the specific neural circuitry engaged by the paradigm and links between striatum and superior temporal areas play a critical role in effective learning.

The contribution of short-term memory for sound features to speech-in-noise perception and cognition

Speech-in-noise (SIN) perception is a fundamental ability that declines with aging, as does general cognition. We assess whether auditory cognitive ability, in particular short-term memory for sound features, contributes to both. We examined how auditory memory for fundamental sound features, the carrier frequency and amplitude modulation rate of modulated white noise, contributes to SIN perception. We assessed SIN in 153 healthy participants with varying degrees of hearing loss using measures that require single-digit perception (the Digits-in-Noise, DIN) and sentence perception (Speech-in-Babble, SIB). Independent variables were auditory memory and a range of other factors including the Pure Tone Audiogram (PTA), a measure of dichotic pitch-in-noise perception (Huggins pitch), and demographic variables including age and sex. Multiple linear regression models were compared using Bayesian Model Comparison. The best predictor model for DIN included PTA and Huggins pitch (r2 = 0.32, p < 0.001), whereas the model for SIB included the addition of auditory memory for sound features (r2 = 0.24, p < 0.001). Further analysis demonstrated that auditory memory also explained a significant portion of the variance (28 %) in scores for a screening cognitive test for dementia. Auditory memory for non-speech sounds may therefore provide an important predictor of both SIN and cognitive ability.

Integrating sound effects and background music in Robotic storytelling – A series of online studies across different story genres

Social robots as storytellers combine advantages of human storytellers – such as embodiment, gestures, and gaze – and audio books – large repertoire of voices, sound effects, and background music. However, research on adding non-speech sounds to robotic storytelling is yet in its infancy. The current series of four online studies investigates the influence of sound effects and background music in robotic storytelling on recipients’ storytelling experience and enjoyment, robot perception, and emotion induction across different story genres, i.e. horror, detective, romantic and humorous stories. Results indicate increased enjoyment for romantic stories and a trend for decreased fatigue for all genres when adding sound effects and background music to the robotic storytelling. Of the four genres examined, horror stories seem to benefit the most from the addition of non-speech sounds. Future research should provide guidelines for the selection of music and sound effects to improve the realization of non-speech sound-accompanied robotic storytelling. In conclusion, our ongoing research suggests that the integration of sound effects and background music holds promise for enhancing robotic storytelling, and our genre comparison provides first guidance of when to use them.

Native language advantage in electrical brain responses to speech sound changes in passive and active listening condition

It is not clear whether the brain can detect changes in native and non-native speech sounds in both unattended and attended conditions, but this information would be important to understand the nature of potential native language advantage in speech perception. We recorded event-related potentials (ERPs) for changes in duration and in Chinese lexical tone in a repeated vowel /a/ in native speakers of Finnish and Chinese in passive and active listening conditions. ERP amplitudes reflecting deviance detection (mismatch negativity; MMN and N2b) and attentional shifts towards changes in speech sounds (P3a and P3b) were investigated. In the passive listening condition, duration changes elicited increased amplitude in the MMN latency window for both standard and deviant sounds in the Finnish speakers compared to the Chinese speakers, but no group differences were observed for P3a. In passive listening to lexical tones, P3a was increased in amplitude for both standard and deviant stimuli in Chinese speakers compared to Finnish speakers, but the groups did not differ in MMN. In active listening, both tone and duration changes elicited N2b and P3b, but the groups differed only in pattern of results for the deviant type. The results thus suggest an overall increased sensitivity to native speech sounds, especially in passive listening, while the mechanisms of change detection and attentional shifting seem to work well for both native and non-native speech sounds in the attentive mode.

Non-speech emotion recognition based on back propagation feed forward networks

Non-speech emotion recognition involves identifying emotions conveyed through non-verbal vocalizations such as laughter, crying, and other sound signals, which play a crucial role in emotional expression and transmission. This paper employs a nine-category discrete emotion model encompassing happy, sad, angry, peaceful, fearful, loving, hateful, brave, and neutral. A proprietary non-speech dataset comprising 2337 instances was utilized, with 384-dimensional feature vectors extracted. The traditional Backpropagation Neural Network (BPNN) algorithm achieved a recognition rate of 87.7% on the non-speech dataset. In contrast, the proposed Whale Optimization Algorithm - Backpropagation Neural Network (WOA-BPNN) algorithm, applied to a self-made non-speech dataset, demonstrated a remarkable accuracy of 98.6%. Notably, even without facial emotional cues, non-speech sounds effectively convey dynamic information, and the proposed algorithm excels in their recognition. The study underscores the importance of non-speech emotional signals in communication, especially with the continuous advancement of artificial intelligence technology. The abstract thus encapsulates the paper’s focus on leveraging AI algorithms for high-precision non-speech emotion recognition.

Ubi-AD : Towards Ubiquitous, Passive Alzheimer Detection using the Smartwatch

Alzheimer’s disease (AD) is a insidious and progressive neurodegenerative disease, the annual relevant social cost for AD patients can reach about $1 trillion in the world. Therefore, early diagnosis and treatment of AD play a vital role in slowing disease progression. However, existing detection methods for cognitive impairment can not consistently screen the stage of AD. To tackle this challenge, we propose an AD detection system, Ubi-AD, which combines the features of multiple biomarkers to realize passive and accurate AD detection. Unlike existing work, Ubi-AD can passively recognize the AD digital biomarkers during daily smartwatch usage without interfering with the user. At the user end, Ubi-AD first extracts the non-speech sounds (pause words, such as em, ah), which contain no privacy-sensitive content. Then, Ubi-AD recognizes the user’s walking activity, dining activity, and sleep activity from daily activities. Ubi-AD analyzes these data from smartwatch and predicts the AD stages using a multi-modal fusion neural network at the cloud end. We evaluate our model on a collected dataset from 45 volunteers. As a result, Ubi-AD can reach a detection accuracy of \(93.4\% \) , which means that Ubi-AD can provide multiple effective biomarkers for ubiquitous and passive detection in daily life.

What do voices contribute to acoustic context effects in speech perception?

All perception takes place in context, and speech perception is no exception. Surrounding sounds form a context that guides speech perception. This is particularly true when earlier sounds inform the perception and/or recognition of later sounds. Several lines of research have advanced models where listeners use preceding context to account for characteristics of the talker’s voice, such as vocal tract properties, articulatory maneuvers, or the speaker’s identity. Conversely, research with nonspeech sounds and/or nonhuman animals has advanced alternative models where perception is responding more to signal acoustics than to vocal characteristics. In this talk, I will review recent evidence from our research on acoustic context effects and talker normalization when hearing speech and nonspeech (music) sounds. Parallel patterns of performance across speech and music domains are conducive to, but not definitive evidence of, acoustic context being utilized according to general signal acoustics, with speech- and voice-specific contributions coming later in the processing stream.

Assessing the role of auditory access in infancy

Even with the widespread adoption of newborn hearing screening and Early Hearing Detection and Intervention systems, infants who are hard-of-hearing (IHH) remain at increased risk for poor or delayed development of auditory and speech perception skills. Speech perception is positively correlated with better auditory development on functional auditory skills and a variety of language outcomes. However, clinical utilization of speech perception tests has not been broadly accepted. One clinically viable tool to assess speech perception during infancy is our electroencephalography (EEG) procedure, which has been validated as an objective measure of speech discrimination. However, we have not examined the development of infant speech perception among IHH and infants with normal hearing (INH). Perceptual attunement—or “narrowing”—is a model of perceptual learning that posits that perceptual abilities are shaped by environmental experiences over the first year of life. Here, we describe our adapted EEG methods to examine the development of infant speech perception to study periods of perceptual attunement between IHH and INH. Preliminary findings confirm that at 3 months of age, infants demonstrate neural encoding of native and non-native speech sounds, and at 6 months of age results suggest a decrease in non-native speech sound encoding.

An exploratory investigation of acoustic features underlying arousal and valence perception of vocalizations from non-speaking individuals

Emotion perception of vocalizations, especially from individuals with no or few spoken words, remains an underexplored topic in acoustical research. The aim of this exploratory study was to identify acoustic features within non-speech vocalizations correlated with perceived arousal and valence. 364 vocalizations were selected from the open-access ReCANVo dataset, comprising non-speech communicative sounds by non-speaking individuals with autism and neurodevelopmental disorders. 108 listeners independently rated each vocalization for arousal and valence on a 5-point Likert scale (78624 total ratings). We then used random forest and elastic net regression techniques to determine correlations between acoustic-based features and perceived affect. Using 4 published (e.g., GeMAPs) and 2 custom feature sets (dim 12, 28) encompassing duration, intensity, pitch, formant, and spectral centroid features, we evaluated each model using R-squared and mean-squared error. Our 12-feature custom elastic net model achieved the highest performance for predicting arousal (R-sq = 0.763, MSE = 0.075), with intensity features weighted highest, comparable to other sets. Our 28-feature random forest model, which included vocal quality features, significantly outperformed other models in valence prediction (R- sq = 0.491; MSE = 0.139, p &amp;lt; 0.001). While further validation on a held- out test set is essential, these exploratory results expand our understanding of affective acoustic features of non-speech sounds from individuals with neurodevelopmental disorders.

Cochlear Implant Users Experience the Sound-To-Music Effect

ABSTRACT The speech-to-song illusion is a robust effect where repeated speech induces the perception of singing; this effect has been extended to repeated excerpts of environmental sounds (sound-to-music effect). Here, we asked whether repetition could elicit musical percepts in cochlear implant (CI) users, who experience challenges with perceiving music due to both physiological and device limitations. Thirty adult CI users and thirty age-matched controls with normal hearing (NH) completed two repetition experiments for speech and nonspeech sounds (water droplets). We hypothesized that CI users would experience the sound-to-music effect from temporal/rhythmic cues alone, but to a lesser magnitude compared to NH controls, given the limited access to spectral information CI users receive from their implants. We found that CI users did experience the sound-to-music effect but to a lesser degree compared to NH participants. Musicality ratings were not associated with musical training or frequency resolution, and among CI users, clinical variables like duration of hearing loss also did not influence ratings. Cochlear implants provide a strong clinical model for disentangling the effects of spectral and temporal information in an acoustic signal; our results suggest that temporal cues are sufficient to perceive the sound-to-music effect when spectral resolution is limited. Additionally, incorporating short repetitions into music specially designed for CI users may provide a promising way for them to experience music.

Effects of Stimulus Rate and Periodicity on Auditory Cortical Entrainment to Continuous Sounds.

The neural mechanisms underlying the exogenous coding and neural entrainment to repetitive auditory stimuli have seen a recent surge of interest. However, few studies have characterized how parametric changes in stimulus presentation alter entrained responses. We examined the degree to which the brain entrains to repeated speech (i.e., /ba/) and nonspeech (i.e., click) sounds using phase-locking value (PLV) analysis applied to multichannel human electroencephalogram (EEG) data. Passive cortico-acoustic tracking was investigated in N = 24 normal young adults utilizing EEG source analyses that isolated neural activity stemming from both auditory temporal cortices. We parametrically manipulated the rate and periodicity of repetitive, continuous speech and click stimuli to investigate how speed and jitter in ongoing sound streams affect oscillatory entrainment. Neuronal synchronization to speech was enhanced at 4.5 Hz (the putative universal rate of speech) and showed a differential pattern to that of clicks, particularly at higher rates. PLV to speech decreased with increasing jitter but remained superior to clicks. Surprisingly, PLV entrainment to clicks was invariant to periodicity manipulations. Our findings provide evidence that the brain's neural entrainment to complex sounds is enhanced and more sensitized when processing speech-like stimuli, even at the syllable level, relative to nonspeech sounds. The fact that this specialization is apparent even under passive listening suggests a priority of the auditory system for synchronizing to behaviorally relevant signals.

Robots’ “Woohoo” and “Argh” Can Enhance Users’ Emotional and Social Perceptions: An Exploratory Study on Non-lexical Vocalizations and Non-linguistic Sounds

As robots have become more pervasive in our everyday life, social aspects of robots have attracted researchers’ attention. Because emotions play a crucial role in social interactions, research has been conducted on conveying emotions via speech. Our study sought to investigate the synchronization of multimodal interaction in human-robot interaction (HRI). We conducted a within-subjects exploratory study with 40 participants to investigate the effects of non-speech sounds (natural voice, synthesized voice, musical sound, and no sound) and basic emotions (anger, fear, happiness, sadness, and surprise) on user perception with emotional body gestures of an anthropomorphic robot (Pepper). While listening to a fairytale with the participant, a humanoid robot responded to the story with recorded emotional non-speech sounds and gestures. Participants showed significantly higher emotion recognition accuracy from the natural voice than from other sounds. The confusion matrix showed that happiness and sadness had the highest emotion recognition accuracy, which is in line with previous research. The natural voice also induced higher trust, naturalness, and preference compared to other sounds. Interestingly, the musical sound mostly showed lower perception ratings, even compared to no sound. Results are discussed with design guidelines for emotional cues from social robots and future research directions.

Hearing aid processing affects acoustic features important for emotional responses to sounds

A wide range of emotional responses is desirable and supports overall quality of life. However, adults with hearing loss exhibit a reduced range of emotional responses to non-speech sounds compared to their peers with normal hearing. Recent work demonstrates that audibility of low- and high-frequencycues supports emotion perception of non-speech sounds, as does modulation energy and roughness in the sound. The purpose of this study was to quantify the effects of hearing aid processing on acoustic parameters that have been related to emotional responses to non-speech sounds. Twenty-three adults with bilateral hearing loss rated valence of non-speech sounds, with and without hearing aids. The non-speech sounds were then recorded through an acoustic manikin with and without the research hearing aids with each participant’s settings. To examine the effects of hearing aid processing, the acoustic parameters previously related to emotional responses were analyzed using a combination of cluster analysis and linear regression. Results indicate that some acoustic parameters are affected by hearing aid processing (e.g., less amplitude modulation at 1270 Hz), which partially explain the reduced range of subjective emotional responses seen in adults with hearing loss. The results of this study have implications for the refinement of hearing aid processing algorithms.

Orthographic influence in the distributional learning of non-native speech sounds

This study investigated the role of orthographic information in the acquisition of non-native speech sounds by monolingual English listeners. Two potentially important orthographic variables were explored: Orthographic compatibility (whether the orthographic information supports or contradicts the distributional information) and orthographic familiarity (whether the native and target languages share the same orthography). Ten groups of learners were trained on either a unimodal or bimodal distribution of two length continua. Out of the 10 groups, eight groups were also exposed to orthographic cues that varied in their compatibility with the distributional information (compatible vs. incompatible) and familiarity with the orthography of learners’ native language (Roman vs. Arabic). Following training, all participants performed an AX discrimination task to test their discrimination of the length contrast. The results revealed that, in general, the availability of either familiar or unfamiliar orthographic input which signaled the existence of a single length category significantly lowered learners’ discrimination of the length contrast regardless of the auditory distribution. Further, the exposure to orthographic input that supported a two-category length distinction enhanced the discrimination of the length contrast irrespective of the distribution. However, the most significant improvement occurred when both distributional information and familiar orthographic input were compatible. Overall, these findings indicate that orthographic input, regardless of its level of compatibility or familiarity, may influence the acquisition of non-native speech sounds.

Cognitive performance in open-plan office acoustic simulations: Effects of room acoustics and semantics but not spatial separation of sound sources

The irrelevant sound effect (ISE) characterizes short-term memory performance impairment during irrelevant sounds relative to quiet. Irrelevant sound presentation in most laboratory-based ISE studies has been rather limited to represent complex scenarios including open-plan offices (OPOs) and not many studies have considered serial recall of heard information. This paper investigates ISE using an auditory-verbal serial recall task, wherein performance was evaluated for relevant factors in simulating OPO acoustics: the irrelevant sounds including the semanticity of speech, reproduction methods over headphones, and room acoustics. Results (Experiments 1 and 2) show that ISE was exhibited in most conditions with anechoic (irrelevant) nonspeech sounds with/without speech, but the effect was substantially higher with meaningful speech compared to foreign speech, suggesting a semantic effect. Performance differences in conditions with diotic and binaural reproductions were not statistically robust, suggesting limited role of spatial separation of sources. In Experiment 3, statistically robust ISE were exhibited for binaural room acoustic conditions with mid-frequency reverberation times, T30 (s) = 0.4, 0.8, 1.1, suggesting cognitive impairment regardless of sound absorption representative of OPOs. Performance differences in T30 = 0.4 s relative to T30 = 0.8 and 1.1 s conditions were statistically robust. This emphasizes the benefits for cognitive performance with increased sound absorption, reinforcing extant room acoustic design recommendations. Performance differences in T30 = 0.8 s vs. 1.1 s were not statistically robust. Collectively, these results suggest that certain findings from ISE studies with idiosyncratic acoustics may not translate well to complex OPO acoustic environments.