Acoustic Context Research Articles

Factors influencing window opening behavior and mechanical ventilation usage during summertime: A case study in UK dwellings

The study presents findings of a data collection campaign conducted in 61 residential buildings in England during the summer of 2022, aiming to investigate the factors influencing 1) window usage, and 2) activation and deactivation of mechanical ventilation systems, and 3) the correlation between the residential soundscape, landscape, individual noise sensitivity, and the significance attributed to the external acoustic environment in determining window usage. The survey, covering 55 dwellings reliant on windows for ventilation, highlights that window opening is predominantly driven by perceived indoor air quality (PIAQ) and thermal comfort concerns. Conversely, the acoustic factor ranks first in prompting window closure, alongside considerations related to perceived cold, safety concerns, excessive drafts, and insect intrusion. Within the 6 dwellings utilizing manually controlled mechanical ventilation, ventilation needs emerges as the predominant factor triggering system activation, followed by considerations related to thermal comfort and PIAQ. Reasons for system deactivation primarily involve a preference for window opening, excessive drafts, and, secondarily, excessive noise or thermal discomfort, with participant-voiced concerns about operational costs. The significance attributed to the external acoustic environment in determining window usage is mainly independent of perceived sound sources, measured loudness, or soundscape content. However, higher acoustic comfort scores correlate with a greater willingness to open windows for an acoustic contact. Interestingly, noise sensitivity modulates the importance attached to the external acoustic context when interacting with the window (odds ratio: 1.04). The study underscores the need of incorporating noise sensitivity as a relevant individual factor in models simulating window closing behaviour.

Speech Prosody Serves Temporal Prediction of Language via Contextual Entrainment.

Temporal prediction assists language comprehension. In a series of recent behavioral studies, we have shown that listeners specifically employ rhythmic modulations of prosody to estimate the duration of upcoming sentences, thereby speeding up comprehension. In the current human magnetoencephalography (MEG) study on participants of either sex, we show that the human brain achieves this function through a mechanism termed entrainment. Through entrainment, electrophysiological brain activity maintains and continues contextual rhythms beyond their offset. Our experiment combined exposure to repetitive prosodic contours with the subsequent presentation of visual sentences that either matched or mismatched the duration of the preceding contour. During exposure to prosodic contours, we observed MEG coherence with the contours, which was source-localized to right-hemispheric auditory areas. During the processing of the visual targets, activity at the frequency of the preceding contour was still detectable in the MEG; yet sources shifted to the (left) frontal cortex, in line with a functional inheritance of the rhythmic acoustic context for prediction. Strikingly, when the target sentence was shorter than expected from the preceding contour, an omission response appeared in the evoked potential record. We conclude that prosodic entrainment is a functional mechanism of temporal prediction in language comprehension. In general, acoustic rhythms appear to endow language for employing the brain's electrophysiological mechanisms of temporal prediction.

UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding

The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing. Audio samples are available at https://cpdu.github.io/unicats.

What do voices contribute to acoustic context effects in speech perception?

All perception takes place in context, and speech perception is no exception. Surrounding sounds form a context that guides speech perception. This is particularly true when earlier sounds inform the perception and/or recognition of later sounds. Several lines of research have advanced models where listeners use preceding context to account for characteristics of the talker’s voice, such as vocal tract properties, articulatory maneuvers, or the speaker’s identity. Conversely, research with nonspeech sounds and/or nonhuman animals has advanced alternative models where perception is responding more to signal acoustics than to vocal characteristics. In this talk, I will review recent evidence from our research on acoustic context effects and talker normalization when hearing speech and nonspeech (music) sounds. Parallel patterns of performance across speech and music domains are conducive to, but not definitive evidence of, acoustic context being utilized according to general signal acoustics, with speech- and voice-specific contributions coming later in the processing stream.

Using a sequence deep learning model to increase the acoustic context of a killer whale detector

Although Killer whales (Orcinus orca) produce many stereotypical vocalizations, their sounds can be difficult to identify in isolation. Experts often rely on acoustic context to accurately identify these animals acoustically. Automated detectors and classifiers, on the other hand, frequently rely on short clips that capture individual vocalizations, not leveraging information regarding previous sounds. We developed deep learning models that used 1-minute inputs containing from 0 to 50 calls, with the average clip having 18. We tested three artificial neural network architectures that used recurrent layers to take the sequence of acoustic events into account. As a baseline, we used a convolutional neural network that only took 3-s clips at a time, without considering sequences of events. Here, we present preliminary evaluations on a dataset containing 360 min with Southern Resident killer whale activity in the Salish Sea, and an equal amount of data without killer whale sounds. The best model used a combination of temporal convolutional layers and gated recurrent units to achieve a recall of 0.95 at the maximum precision of 0.98. The models will be applied to near real-time monitoring efforts and will be open-sourced in the future.

The effects of variability on context effects and psychometric function slopes in speaking rate normalizationa).

Acoustic context influences speech perception, but contextual variability restricts this influence. Assgari and Stilp [J. Acoust. Soc. Am. 138, 3023-3032 (2015)] demonstrated that when categorizing vowels, variability in who spoke the preceding context sentence on each trial but not the sentence contents diminished the resulting spectral contrast effects (perceptual shifts in categorization stemming from spectral differences between sounds). Yet, how such contextual variability affects temporal contrast effects (TCEs) (also known as speaking rate normalization; categorization shifts stemming from temporal differences) is unknown. Here, stimuli were the same context sentences and conditions (one talker saying one sentence, one talker saying 200 sentences, 200 talkers saying 200 sentences) used in Assgari and Stilp [J. Acoust. Soc. Am. 138, 3023-3032 (2015)], but set to fast or slow speaking rates to encourage perception of target words as "tier" or "deer," respectively. In Experiment 1, sentence variability and talker variability each diminished TCE magnitudes; talker variability also produced shallower psychometric function slopes. In Experiment 2, when speaking rates were matched across the 200-sentences conditions, neither TCE magnitudes nor slopes differed across conditions. In Experiment 3, matching slow and fast rates across all conditions failed to produce equal TCEs and slopes everywhere. Results suggest a complex interplay between acoustic, talker, and sentence variability in shaping TCEs in speech perception.

Musical training is not associated with spectral context effects in instrument sound categorization.

Musicians display a variety of auditory perceptual benefits relative to people with little or no musical training; these benefits are collectively referred to as the "musician advantage." Importantly, musicians consistently outperform nonmusicians for tasks relating to pitch, but there are mixed reports as to musicians outperforming nonmusicians for timbre-related tasks. Due to their experience manipulating the timbre of their instrument or voice in performance, we hypothesized that musicians would be more sensitive to acoustic context effects stemming from the spectral changes in timbre across a musical context passage (played by a string quintet then filtered) and a target instrument sound (French horn or tenor saxophone; Experiment 1). Additionally, we investigated the role of a musician's primary instrument of instruction by recruiting French horn and tenor saxophone players to also complete this task (Experiment 2). Consistent with the musician advantage literature, musicians exhibited superior pitch discrimination to nonmusicians. Contrary to our main hypothesis, there was no difference between musicians and nonmusicians in how spectral context effects shaped instrument sound categorization. Thus, musicians may only outperform nonmusicians for some auditory skills relevant to music (e.g., pitch perception) but not others (e.g., timbre perception via spectral differences).

Response generation in multi-modal dialogues with split pre-generation and cross-modal contrasting

Due to the natural multi-modal occurrence format (text, audio, vision) of the dialogues, textual response generation in dialogues should rely on the multi-modal contexts beyond text only. However, most existing studies normally ignore the rich information of other modalities, such as audio. To investigate the importance of the acoustic contexts, we explore the multi-modal dialogue scenario with aligned text and audio temporal sequences for textual response generation of an assumed system, namely RGMD task. To this end, we construct a new multi-modal dataset for this task based on TV shows, which contains 84.9K utterances. Considering the response diversity limited by the context and modality interactions for RGMD, we attempt the split pre-generation (SPG) strategy and the cross-modal contrastive learning (CCL) strategy in multi-modal pre-training for better response generation. On the one hand, with SPG, we can obtain many diverse responses without the restrictions of too many historical mixed multi-modal contexts. On the other hand, with CCL, we can capture the interactions between text and audio. Extensive experiments demonstrate that our approach based on BART can consistently perform better than the state-of-the-art textual approach DP by 4.17%, 8.96%, 2.43%, 1.04% and 7.54% regarding metrics of BLEU, DIST, ROUGE, METEOR and NIST, respectively. Moreover, our approach based on GPT can outperform the state-of-the-art multi-modal approach RLM by 6.79%, 9.25%, 7.49%, 9.31% and 13.75% regarding metrics of BLEU, DIST, ROUGE, METEOR and NIST, respectively. Besides, we conduct much in-depth analysis, showing the necessity of audio for response generation and further verifying the effectiveness of our approach.

Internal Cognitive Load Differentially Influences Acoustic and Lexical Context Effects in Speech Perception: Evidence From a Population With Attention-Deficit/Hyperactivity Disorder.

To overcome variability in spoken language, listeners utilize various types of context information for disambiguating speech sounds. Context effects have been shown to be affected by cognitive load. However, previous results are mixed regarding the influence of cognitive load on the use of context information in speech perception. We tested a population characterized by an attention-deficit/hyperactivity disorder (ADHD) to better understand the relationship between attention (or internal cognitive load) and context effects. The use of acoustic versus lexical properties of the surrounding signal to disambiguate speech sounds was examined in listeners with ADHD and neurotypical listeners. Compared to neurotypicals, individuals with ADHD relied more strongly on lexical context for speech perception; however, reliance on acoustic context information from speech rate did not differ. These findings confirm that cognitive load impacts the use of high-level but not low-level context information in speech and imply that speech recognition deficits in ADHD likely arise due to impaired higher order cognitive processes.

Analysis of the Performance of IoT Networks in Acoustic Environment by using LZW Data Compression Technique

The Internet of Things (IoT) has experienced phenomenal growth, opening up a wide range of applications in many settings. Due to the properties of sound propagation, IoT networks operating in acoustic environments in particular present special difficulties. Data compression techniques can be used to minimize overhead and maximize resource utilization in these networks to increase performance. The performance of IoT networks in acoustic environments is examined in this study, with a focus on routing overhead, throughput, and typical end-to-end delay. Lempel-Ziv-Welch (LZW) data compression is used to reduce data size and boost communication effectiveness. Three well-known protocols—MQTT, CoAP, and Machine-to-Machine (M2M)—are assessed in relation to acoustic Internet of Things networks. To mimic different acoustic conditions and collect performance metrics, a thorough experimental setup is used. Different network topologies, data speeds, and compression settings are used in the studies to determine how they affect the performance metrics. According to the analysis's findings, all three protocols' routing overhead is greatly decreased by the LZW data compression approach, which enhances network scalability and lowers energy usage. Additionally, the compressed data size has a positive impact on network throughput, allowing for effective data transmission in acoustic contexts with limited resources. Additionally, using LZW compression is seen to minimize the average end-to-end delay, improving real-time communication applications. This study advances knowledge of IoT networks operating in acoustic environments and the effects of data reduction methods on their functionality. The results offer useful information for network engineers and system designers to optimize the performance of IoT networks in similar situations. Additionally, a comparison of the MQTT, CoAP, and M2M protocols' suitability for acoustic IoT deployments is provided, assisting in the choice of protocol for particular application needs.

A study of low frequency sound insulation mechanism of a perforated plate-type acoustic metamaterial

This paper presents a new design of a perforated plate-type acoustic metamaterial (PAM) to evade the influence of membrane pre-tension on the acoustic performance of metamaterials and to enable air ventilation while achieving the designated sound insulation. The study embarks on the analysis of sound insulation mechanism of a typical perforated membrane-type acoustic metamaterial (MAM) from the acoustic context, and discloses that sound transmission loss (STL) peaks of a perforated MAM are due to the strong wave interference between sound sources transmitted through the orifice(s) and the membrane(s). The sound insulation mechanism is explored further in the study by an acoustic impedance analysis of a perforated metamaterial using an electro-acoustic analogy in the latter part of the paper, and the result confirms that the STL peaks of a perforated PAM (or MAM) are indeed the outcome of a strong sound interference. The study also compares an experiment test and a finite element simulation on the sound transmission of a perforated PAM based on the new design. The result confirms that the new perforated PAM can yield a good broadband STL at relatively low frequencies. A practical application using the perforated PAM design is applied to reduce the noise propagation from the compressor compartment of a commercial refrigerator. The test result verifies the effectiveness of the design in terms of both sound reduction and air ventilation.

A neuron model with unbalanced synaptic weights explains the asymmetric effects of anaesthesia on the auditory cortex.

Substantial progress in the field of neuroscience has been made from anaesthetized preparations. Ketamine is one of the most used drugs in electrophysiology studies, but how ketamine affects neuronal responses is poorly understood. Here, we used in vivo electrophysiology and computational modelling to study how the auditory cortex of bats responds to vocalisations under anaesthesia and in wakefulness. In wakefulness, acoustic context increases neuronal discrimination of natural sounds. Neuron models predicted that ketamine affects the contextual discrimination of sounds regardless of the type of context heard by the animals (echolocation or communication sounds). However, empirical evidence showed that the predicted effect of ketamine occurs only if the acoustic context consists of low-pitched sounds (e.g., communication calls in bats). Using the empirical data, we updated the naïve models to show that differential effects of ketamine on cortical responses can be mediated by unbalanced changes in the firing rate of feedforward inputs to cortex, and changes in the depression of thalamo-cortical synaptic receptors. Combined, our findings obtained in vivo and in silico reveal the effects and mechanisms by which ketamine affects cortical responses to vocalisations.

Statistical Pass-By for Unattended Road Traffic Noise Measurement in an Urban Environment.

Low-noise surfaces have become a common mitigation action in the last decade, so much so that different methods for feature extraction have been established to evaluate their efficacy. Among these, the Close Proximity Index (CPX) evaluates the noise emissions by means of multiple runs at different speeds performed with a vehicle equipped with a reference tire and with acoustic sensors close to the wheel. However, signals acquired with CPX make it source oriented, and the analysis does not consider the real traffic flow of the studied site for a receiver-oriented approach. These aspects are remedied by Statistical Pass-By (SPB), a method based on sensor feature extraction with live detection of events; noise and speed acquisitions are performed at the roadside in real case scenarios. Unfortunately, the specific SPB requirements for its measurement setup do not allow an evaluation in urban context unless a special setup is used, but this may alter the acoustical context in which the measurement was performed. The present paper illustrates the testing and validation of a method named Urban Pass-By (U-SPB), developed during the LIFE NEREiDE project. U-SPB originates from standard SPB, exploits unattended measurements and develops an in-lab feature detection and extraction procedure. The U-SPB extends the evaluation in terms of before/after data comparison of the efficiency of low-noise laying in an urban context while combining the estimation of long-term noise levels and traffic parameters for other environmental noise purposes, such as noise mapping and action planning.

Talker and sentence variability attenuates speaking rate normalization in speech perception

Speech perception is shaped by acoustic context effects, where acoustic properties of earlier (context) sounds influence categorization of later (target) sounds. These context effects are sensitive to talker characteristics. When context sentences were spoken by a different talker on each trial, spectral contrast effect (SCE) magnitudes affecting categorization of the target vowel were smaller than when sentences were spoken by a single talker [Assgari and Stilp, J. Ascoust. Soc. Am. (2015)]. Here, perceptual consequences of talker variability were examined in Temporal Contrast Effects (TCEs; also termed rate normalization). Each trial presented a context sentence (spoken at a fast or slow rate) followed by a target word varying from “deer” to “tier.” Context sentences were the same stimuli from Assgari and Stilp (2015): one talker speaking the same sentence on each trial, one talker speaking a different sentence on each trial, and 200 talkers each speaking a different sentence on each trial. TCE magnitudes significantly decreased between One Talker/One Sentence and One Talker/200 Sentences, and again between One Talker/200 Sentences and 200 Talkers/200 Sentences. Conversely, in Assgari and Stilp, talker variability diminished SCEs but sentence variability for a single talker did not. These results suggest that variability in different acoustic domains restricts speech perception.

AIDA: An Active Inference-Based Design Agent for Audio Processing Algorithms

In this paper we present Active Inference-Based Design Agent (AIDA), which is an active inference-based agent that iteratively designs a personalized audio processing algorithm through situated interactions with a human client. The target application of AIDA is to propose on-the-spot the most interesting alternative values for the tuning parameters of a hearing aid (HA) algorithm, whenever a HA client is not satisfied with their HA performance. AIDA interprets searching for the “most interesting alternative” as an issue of optimal (acoustic) context-aware Bayesian trial design. In computational terms, AIDA is realized as an active inference-based agent with an Expected Free Energy criterion for trial design. This type of architecture is inspired by neuro-economic models on efficient (Bayesian) trial design in brains and implies that AIDA comprises generative probabilistic models for acoustic signals and user responses. We propose a novel generative model for acoustic signals as a sum of time-varying auto-regressive filters and a user response model based on a Gaussian Process Classifier. The full AIDA agent has been implemented in a factor graph for the generative model and all tasks (parameter learning, acoustic context classification, trial design, etc.) are realized by variational message passing on the factor graph. All verification and validation experiments and demonstrations are freely accessible at our GitHub repository.

Study of Built Acoustic Context on the Psychological Response of Young Adults in Urban Green Spaces

Study of Built Acoustic Context on the Psychological Response of Young Adults in Urban Green Spaces

A machine learning pipeline for classification of cetacean echolocation clicks in large underwater acoustic datasets.

Machine learning algorithms, including recent advances in deep learning, are promising for tools for detection and classification of broadband high frequency signals in passive acoustic recordings. However, these methods are generally data-hungry and progress has been limited by challenges related to the lack of labeled datasets adequate for training and testing. Large quantities of known and as yet unidentified broadband signal types mingle in marine recordings, with variability introduced by acoustic propagation, source depths and orientations, and interacting signals. Manual classification of these datasets is unmanageable without an in-depth knowledge of the acoustic context of each recording location. A signal classification pipeline is presented which combines unsupervised and supervised learning phases with opportunities for expert oversight to label signals of interest. The method is illustrated with a case study using unsupervised clustering to identify five toothed whale echolocation click types and two anthropogenic signal categories. These categories are used to train a deep network to classify detected signals in either averaged time bins or as individual detections, in two independent datasets. Bin-level classification achieved higher overall precision (>99%) than click-level classification. However, click-level classification had the advantage of providing a label for every signal, and achieved higher overall recall, with overall precision from 92 to 94%. The results suggest that unsupervised learning is a viable solution for efficiently generating the large, representative training sets needed for applications of deep learning in passive acoustics.

Acoustic Context Modulates Natural Sound Discrimination in Auditory Cortex through Frequency-Specific Adaptation.

Sound discrimination is essential in many species for communicating and foraging. Bats, for example, use sounds for echolocation and communication. In the bat auditory cortex there are neurons that process both sound categories, but how these neurons respond to acoustic transitions, that is, echolocation streams followed by a communication sound, remains unknown. Here, we show that the acoustic context, a leading sound sequence followed by a target sound, changes neuronal discriminability of echolocation versus communication calls in the cortex of awake bats of both sexes. Nonselective neurons that fire equally well to both echolocation and communication calls in the absence of context become category selective when leading context is present. On the contrary, neurons that prefer communication sounds in the absence of context turn into nonselective ones when context is added. The presence of context leads to an overall response suppression, but the strength of this suppression is stimulus specific. Suppression is strongest when context and target sounds belong to the same category, e.g.,echolocation followed by echolocation. A neuron model of stimulus-specific adaptation replicated our results in silico The model predicts selectivity to communication and echolocation sounds in the inputs arriving to the auditory cortex, as well as two forms of adaptation, presynaptic frequency-specific adaptation acting in cortical inputs and stimulus-unspecific postsynaptic adaptation. In addition, the model predicted that context effects can last up to 1.5 s after context offset and that synaptic inputs tuned to low-frequency sounds (communication signals) have the shortest decay constant of presynaptic adaptation.SIGNIFICANCE STATEMENT We studied cortical responses to isolated calls and call mixtures in awake bats and show that (1) two neuronal populations coexist in the bat cortex, including neurons that discriminate social from echolocation sounds well and neurons that are equally driven by these two ethologically different sound types; (2) acoustic context (i.e., other natural sounds preceding the target sound) affects natural sound selectivity in a manner that could not be predicted based on responses to isolated sounds; and (3) a computational model similar to those used for explaining stimulus-specific adaptation in rodents can account for the responses observed in the bat cortex to natural sounds. This model depends on segregated feedforward inputs, synaptic depression, and postsynaptic neuronal adaptation.

Video sentiment analysis with bimodal information-augmented multi-head attention

Humans express feelings or emotions via different channels. Take language as an example, it entails different sentiments under different visual–acoustic contexts. To precisely understand human intentions as well as reduce the misunderstandings caused by ambiguity and sarcasm, we should consider multimodal signals including textual, visual and acoustic signals. The crucial challenge is to fuse different modalities of features for sentiment analysis. To effectively fuse the information carried by different modalities and better predict the sentiments, we design a novel multi-head attention based fusion network, which is inspired by the observations that the interactions between any two pair-wise modalities are different and they do not equally contribute to the final sentiment prediction. By assigning the acoustic–visual, acoustic–textual and visual–textual features with reasonable attention and exploiting a residual structure, we attend to attain the significant features. We conduct extensive experiments on four public multimodal datasets including one in Chinese and three in English. The results show that our approach outperforms the existing methods and can explain the contributions of bimodal interaction in multiple modalities.

Groove Fields. Understanding the Dance Floor from an Art-Based Research Perspective

This contribution examines groove dynamics across various spatial contexts such as the dance club and the artistic performance space. As groove dynamics do not occur in a spatial vacuum but are embedded in affective environments, they need to be approached from an interdisciplinary perspective that ties together findings from experimental field studies, performance studies, musicology, cognitive hermeneutics etc. and places them in the physical, anatomic and acoustic contexts of concrete dance situations. In that way, the dance floor can be understood as a space of entanglements where acoustic, movement and kinaesthetic grooves mark and create a fluid territory of improvisational interaction and communication.