Sound Categories Research Articles

Dolphin self-talk: unusual acoustic behaviour of a solitary bottlenose dolphin

ABSTRACT Solitary dolphins occurring outside their normal range provide a unique opportunity to study acoustic signals in the complete absence of conspecifics, drawing insights into their communicative functions. In this study, we analysed the acoustic behaviour of a solitary bottlenose dolphin residing in the Baltic Sea. We expected that he would produce few, if any, communicative sounds in the absence of potential recipients. Contrary to this expectation, we found the dolphin to be highly vocal, emitting burst-pulse and tonal sounds in rhythmic bouts. The sounds did not co-occur with echolocation trains, indicating that they were not produced during feeding activities, despite the presence of bray calls previously identified as feeding-associated in this population. The dolphin produced three different types of stereotyped whistles, deviating from the typical pattern of one individually distinctive signature whistle per dolphin. Additionally, three different biphonic sound categories were identified: combinations of two whistle components, a whistle and a low-frequency tonal component, and two burst-pulse components. In conclusion, the production of vocalisations in the absence of conspecifics suggests that these sounds may be emitted unintentionally as emotional signals, serve functions other than direct communication, or simply be a byproduct of dolphins’ intrinsic need for social interaction.

Individual differences in the perception of phonetic category structure predict speech-in-noise performance.

Speech sounds exist in a complex acoustic-phonetic space, and listeners vary in the extent to which they are sensitive to variability within the speech sound category ("gradience") and the degree to which they show stable, consistent responses to phonetic stimuli. Here, we investigate the hypothesis that individual differences in the perception of the sound categories of one's language may aid speech-in-noise performance across the adult lifespan. Declines in speech-in-noise performance are well documented in healthy aging, and are, unsurprisingly, associated with differences in hearing ability. Nonetheless, hearing status and age are incomplete predictors of speech-in-noise performance, and long-standing research suggests that this ability draws on more complex cognitive and perceptual factors. In this study, a group of adults ranging in age from 18 to 67 years performed online assessments designed to measure phonetic category sensitivity, questionnaires querying recent noise exposure history and demographic factors, and crucially, a test of speech-in-noise perception. Results show that individual differences in the perception of two consonant contrasts significantly predict speech-in-noise performance, even after accounting for age and recent noise exposure history. This finding supports the hypothesis that individual differences in sensitivity to phonetic categories mediates speech perception in challenging listening situations.

When Jack isn't Jacques: Simultaneous opposite language-specific speech perceptual learning in French-English bilinguals.

Humans are remarkably good at understanding spoken language, despite the huge variability of the signal as a function of the talker, the situation, and the environment. This success relies on having access to stable representations based on years of speech input, coupled with the ability to adapt to short-term deviations from these norms, e.g. accented speech or speech altered by ambient noise. In the last two decades, there has been a robust research effort focused on a possible mechanism for adjusting to accented speech. In these studies, listeners typically hear 15 - 20 words in which a speech sound has been altered, creating a short-term deviation from its longer-term representation. After exposure to these items, listeners demonstrate "lexically driven phonetic recalibration"-they alter their categorization of speech sounds, expanding a speech category to take into account the recently heard deviations from their long-term representations. In the current study, we investigate such adjustments by bilingual listeners. French-English bilinguals were first exposed to nonstandard pronunciations of a sound (/s/ or /f/) in one language and tested for recalibration in both languages. Then, the exposure continued with both the original type of mispronunciation in the same language, plus mispronunciations in the other language, in the opposite direction. In a final test, we found simultaneous recalibration in opposite directions for the two languages-listeners shifted their French perception in one direction and their English in the other: Bilinguals can maintain separate adjustments, for the same sounds, when a talker's speech differs across two languages.

Can the Mismatch Negativity Really Be Elicited by Abstract Linguistic Contrasts?

Abstract The mismatch negativity (MMN) is an event-related potential component that reflects pre-attentive change detection in the brain. As an electrophysiological index of processing that responds to differences in incoming consecutive stimuli, the MMN can be elicited through, for example, the presentation of two different categories of sounds in an oddball paradigm where sounds from the “standard” category occur frequently and sounds from the “deviant” category occur rarely. The specificity of what can elicit the MMN is yet to be fully defined. Here we test whether the MMN can be generated by an abstract linguistic contrast with no reliable acoustic cue. Previous studies have shown that the way in which an acoustic cue is used to elicit MMN is influenced by linguistic knowledge, but have not shown that a nonacoustic, abstract linguistic contrast can itself elicit MMN. In this study, we test the strongest interpretation of the claim that the MMN can be generated through a purely linguistic contrast by contrasting tenses in ablauting irregular English verbs (where there is no reliable acoustic cue for tense). We find that this contrast elicits a negativity, as do other linguistic contrasts previously shown to elicit MMN (a contrast between phonologically voiced and phonologically voiceless segments and a purely acoustic contrast between aspirated and unaspirated segments). The findings provide evidence that the MMN is indeed sensitive to purely abstract linguistic categories.

Impact of interference on vocal and instrument recognition.

Voices arguably occupy a superior role in auditory processing. Specifically, studies have reported that singing voices are processed faster and more accurately and possess greater salience in musical scenes compared to instrumental sounds. However, the underlying acoustic features of this superiority and the generality of these effects remain unclear. This study investigates the impact of frequency micro-modulations (FMM) and the influence of interfering sounds on sound recognition. Thirty young participants, half with musical training, engage in three sound recognition experiments featuring short vocal and instrumental sounds in a go/no-go task. Accuracy and reaction times are measured for sounds from recorded samples and excerpts of popular music. Each sound is presented in separate versions with and without FMM, in isolation or accompanied by a piano. Recognition varies across sound categories, but no general vocal superiority emerges and no effects of FMM. When presented together with interfering sounds, all sounds exhibit degradation in recognition. However, whereas /a/ sounds stand out by showing a distinct robustness to interference (i.e., less degradation of recognition), /u/ sounds lack this robustness. Acoustical analysis implies that recognition differences can be explained by spectral similarities. Together, these results challenge the notion of general vocal superiority in auditory perception.

Developing Bluetooth phonocardiogram for detecting heart murmurs using hybrid MFCC and LSTM

Cardiovascular disease is a leading global cause of mortality. Most stethoscopes still necessitate the use of tubing, which entails direct physical contact between the healthcare provider and patient. The stethoscope can serve as a means of transmission if it is utilized on individuals who have been diagnosed with airborne and droplet-borne infectious illnesses. A prototype was created to capture heart sounds using a Phonocardiography (PCG) device over website-based Bluetooth connectivity. This approach offers the benefits of being cost-effective, facilitating computer-aided diagnostics, and being wearable. In addition, the primary significance of this study resides in the identification of heart sound irregularities caused by cardio dynamic abnormalities of the heart valves, known as murmurs. The heart sound categorization process utilizes a machine learning model that involves extracting 25 Mel frequency cepstral coefficients (MFCC) as features. The model employs a hybrid approach combining convolutional neural network and long short-term memory (CNN-LSTM) techniques. The research findings indicate that the suggested model achieves an average accuracy rate of 95.9% over five distinct categories, i.e., normal, atrial stenosis, mitral regurgitation, mitral stenosis, and mitral valves prolapse. Further study can be conducted on hardware development by incorporating an infrared sensor at the fingertip of the stethoscope.

People follow motivation-structural rules when they react to synthetised sounds

Emotionally expressive vocalizations can elicit approach-avoidance responses in humans and non-human animals. We investigated whether artificially generated sounds have similar effects on humans. We assessed whether subjects' reactions were linked to acoustic properties, and associated valence and intensity. We generated 343 artificial sounds with differing call lengths, fundamental frequencies and added acoustic features across 7 categories and 3 levels of biological complexity. We assessed the hypothetical behavioural response using an online questionnaire with a manikin task, in which 172 participants indicated whether they would approach or withdraw from an object emitting the sound. (1) Quieter sounds elicited approach, while loud sounds were associated with avoidance. (2) The effect of pitch was modulated by category, call length and loudness. (2a) Low-pitched sounds in complex sound categories prompted avoidance, while in other categories they elicited approach. (2b) Higher pitch in loud sounds had a distancing effect, while higher pitch in quieter sounds prompted approach. (2c) Longer sounds promoted avoidance, especially at high frequencies. (3) Sounds with higher intensity and negative valence elicited avoidance. We conclude that biologically based acoustic signals can be used to regulate the distance between social robots and humans, which can provide an advantage in interactive scenarios.

Neural correlates of phonetic categorization under auditory (phoneme) and visual (grapheme) modalities.

We tested whether the neural mechanisms of phonetic categorization are specific to speech sounds or generalize to graphemes (i.e., visual letters) of the same phonetic label. Given that linguistic experience shapes categorical processing, and letter-speech sound matching plays a crucial role during early reading acquisition, we hypothesized sound phoneme and visual grapheme tokens representing the same linguistic identity might recruit common neural substrates, despite originating from different sensory modalities. Behavioral and neuroelectric brain responses (ERPs) were acquired as participants categorized stimuli from sound (phoneme) and homologous letter (grapheme) continua each spanning a /da/ - /ga/ gradient. Behaviorally, listeners were faster and showed stronger categorization of phoneme compared to graphemes. At the neural level, multidimensional scaling of the EEG revealed responses self-organized in a categorial fashion such that tokens clustered within their respective modality beginning ∼150-250 ms after stimulus onset. Source-resolved ERPs further revealed modality-specific and overlapping brain regions supporting phonetic categorization. Left inferior frontal gyrus and auditory cortex showed stronger responses for sound category members compared to phonetically ambiguous tokens, whereas early visual cortices paralleled this categorical organization for graphemes. Auditory and visual categorization also recruited common visual association areas in extrastriate cortex but in opposite hemispheres (auditory = left; visual=right). Our findings reveal both auditory and visual sensory cortex supports categorical organization for phonetic labels within their respective modalities. However, a partial overlap in phoneme and grapheme processing among occipital brain areas implies the presence of an isomorphic, domain-general mapping for phonetic categories in dorsal visual system.

A hierarchy of processing complexity and timescales for natural sounds in human auditory cortex.

Efficient behavior is supported by humans' ability to rapidly recognize acoustically distinct sounds as members of a common category. Within auditory cortex, there are critical unanswered questions regarding the organization and dynamics of sound categorization. Here, we performed intracerebral recordings in the context of epilepsy surgery as 20 patient-participants listened to natural sounds. We built encoding models to predict neural responses using features of these sounds extracted from different layers within a sound-categorization deep neural network (DNN). This approach yielded highly accurate models of neural responses throughout auditory cortex. The complexity of a cortical site's representation (measured by the depth of the DNN layer that produced the best model) was closely related to its anatomical location, with shallow, middle, and deep layers of the DNN associated with core (primary auditory cortex), lateral belt, and parabelt regions, respectively. Smoothly varying gradients of representational complexity also existed within these regions, with complexity increasing along a posteromedial-to-anterolateral direction in core and lateral belt, and along posterior-to-anterior and dorsal-to-ventral dimensions in parabelt. When we estimated the time window over which each recording site integrates information, we found shorter integration windows in core relative to lateral belt and parabelt. Lastly, we found a relationship between the length of the integration window and the complexity of information processing within core (but not lateral belt or parabelt). These findings suggest hierarchies of timescales and processing complexity, and their interrelationship, represent a functional organizational principle of the auditory stream that underlies our perception of complex, abstract auditory information.

Sound event detection in traffic scenes based on graph convolutional network to obtain multi-modal information

Sound event detection involves identifying sound categories in audio and determining when they start and end. However, in real-life situations, sound events are usually not isolated. When one sound event occurs, there are often other related sound events that take place as co-occurrences or successive occurrences. The timing relationship of sound events can reflect their characteristics. Therefore, this paper proposes a sound event detection method for traffic scenes based on a graph convolutional network, which considers this timing relationship as a form of multimodal information. The proposed method involves using the acoustic event window method to obtain co-occurrences or successive occurrences of relationship information in the sound signal while filtering out possible noise relationship information. This information is then represented as a graphical structure. Next, the graph convolutional neural network is improved to balance relationship weights between neighbors and itself and to avoid excessive smoothing. It is used to learn the relationship information in the graph structure. Finally, the convolutional recurrent neural network is used to learn the acoustic feature information of sound events, and the relationship information of sound events is obtained by multi-modal fusion to enhance the performance of sound event detection. The experimental results show that using multi-modal information with the proposed method can effectively improve the performance of the model and enhance the perception ability of smart cars in their surrounding environment while driving.

The incidence of bird sounds, and other categories of non-focal sounds, confound the relationships between acoustic indices and bird species richness in southern China

The analysis of audio recordings through acoustic indices has been proposed as an efficient way to measure and monitor biodiversity, given the assumption that higher levels of biodiversity produce more rapidly-changing and complex sound. However, in previous work in south China, we have found only moderate correlations between the acoustic indices and bird species richness, when analyzing recordings that were made at the same time as conducting point counts of birds. Here, we extended that work in three study regions in Guangxi Province, making observations both inside relatively undisturbed forest reserves and in the surrounding agricultural lands. We found that of 42 correlations between bird species richness and the acoustic indices (seven commonly used acoustic indices [ACI, ADI, AEI, AR, BIO, H, NDSI] each calculated in six habitat/region combinations) only seven had even a moderate relationship (i.e., |r| ≥ 0.20). To understand this paucity of relationships, we listened to a subsample of the recordings, scoring them for various kinds of biophony (incidence of bird and insect sounds), anthropophony (incidence of observer-produced sounds, sounds made by other humans, and sounds made by machines) and geophony (primarily wind, occasionally water). These analyses found that insects were positively related to the acoustic indices, especially in forests. Moreover, anthropophony and geophony, were, in general, negatively correlated to the indices. The incidence of bird vocalizations also had consistent and strong correlations to the residuals of the models relating the acoustic indices to bird species richness, demonstrating that the indices are clearly sensitive to the amount of bird sounds, but that is not necessarily strongly correlated to bird diversity. Multivariate models confirmed that the amounts of different sound categories were more influential on the acoustic indices than bird species richness. Our results demonstrate how competing noises, in both relatively undisturbed environments and heavily modified ones, can confound relationships between the acoustic indices and biodiversity. We discuss possible remedial steps for using acoustic indices in noisy soundscapes, including tailoring the selection of indices and frequency bands for the objectives of the project, or, alternatively, using a big-data approach that combines multiple indices.

Normative ratings for the Kitchen and Food Sounds (KFS) database

Sounds are important sensory cues for food perception and acceptance. We developed and validated a large-scale database of kitchen and food sounds (180 stimuli) capturing different stages of preparing, cooking, serving, and/or consuming foods and beverages and sounds of packaging, kitchen utensils, and appliances. Each sound was evaluated across nine subjective evaluative dimensions (random order), including stimuli-related properties (e.g., valence, arousal) and food-related items (e.g., healthfulness, appetizingness) by a subsample of 51 to 64 participants (Mdn = 54; N = 332; 69.6% women, Mage = 27.46 years, SD = 10.20). Participants also identified each sound and rated how confident they were in such identification. Results show that, overall, participants could correctly identify the sound or at least recognize the general sound categories. The stimuli of the KFS database varied across different levels (low, moderate, high) of the evaluative dimensions under analysis, indicating good adequacy to a broad range of research purposes. The correlation analysis showed a high degree of association between evaluative dimensions. The sociodemographic characteristics of the sample had a limited influence on the stimuli evaluation. Still, some aspects related to food and cooking were associated with how the sounds are evaluated, suggesting that participants’ proficiency in the kitchen should be considered when planning studies with food sounds. Given its broad range of stimulus categories and evaluative dimensions, the KFS database (freely available at OSF) is suitable for different research domains, from fundamental (e.g., cognitive psychology, basic sensory science) to more applied research (e.g., marketing, consumer science).

The Effect of Indian Contact and Glaswegian Contact on the Phonetic Backward Transfer of Glaswegian English (L2) on Hindi and Indian English (L1)

This study examined phonetic backward transfer in ‘Glaswasians’, the ethnolinguistic minority of first-generation bilingual immigrant Indians in Glasgow (Scotland), who present a situation of contact between their native languages of Hindi and Indian English (L1s) and the dominant host language and dialect, Glaswegian English (L2). This was examined in relation to the Revised Speech Learning Model (SLM-r) and Speech Accommodation Framework. These predict that the migrants’ L1 sound categories can either shift to become more Glaswegian-like (‘assimilation’ or ‘convergence’) or exaggeratedly Indian-like (‘dissimilation’ or ‘divergence’) or remain unchanged. The effect of Indian and Glaswegian Contact on transfer was also investigated. Two control groups (Indians and Glaswegians) and the experimental group (Glaswasians) were recorded reading English and Hindi sentences containing multiple phones which were examined for multiple phonetic features (/t/—VOT, /l/—F2-F1 difference, /b d g/—Relative Burst Intensity). In both languages, Glaswasian /t/ and /g/ became more Glaswegian-like (assimilation), whereas F2-F1 difference in /l/ became exaggeratedly Indian-like (dissimilation). Higher Indian Contact was associated with more native-like values in /t/ and /l/ in Hindi but had no influence on /g/. Higher Glaswegian Contact was related to increased assimilation of /g/ in English but had no effect on /l/ and /t/.

Advancing Cough Classification: Swin Transformer vs. 2D CNN with STFT and Augmentation Techniques

Coughing, a common symptom associated with various respiratory problems, is a crucial indicator for diagnosing and tracking respiratory diseases. Accurate identification and categorization of cough sounds, specially distinguishing between wet and dry coughs, are essential for understanding underlying health conditions. This research focuses on applying the Swin Transformer for classifying wet and dry coughs using short-time Fourier transform (STFT) representations. We conduct a comprehensive evaluation, including a performance comparison with a 2D convolutional neural network (2D CNN) model, and exploration of two distinct image augmentation methods: time mask augmentation and classical image augmentation techniques. Extensive hyperparameter tuning is performed to optimize the Swin Transformer’s performance, considering input size, patch size, embedding size, number of epochs, optimizer type, and regularization technique. Our results demonstrate the Swin Transformer’s superior accuracy, particularly when trained on classically augmented STFT images with optimized settings (320 × 320 input size, RMS optimizer, 8 × 8 patch size, and an embedding size of 128). The approach achieves remarkable testing accuracy (88.37%) and ROC AUC values (94.88%) on the challenging crowdsourced COUGHVID dataset, marking improvements of approximately 2.5% and 11% increases in testing accuracy and ROC AUC values, respectively, compared to previous studies. These findings underscore the efficacy of Swin Transformer architectures in disease detection and healthcare classification problems.

Speech Sound Categories Affect Lexical Competition: Implications for Analytic Auditory Training.

We provide a novel application of psycholinguistic theories and methods to the field of auditory training to provide preliminary data regarding which minimal pair contrasts are more difficult for listeners with typical hearing to distinguish in real-time. Using eye-tracking, participants heard a word and selected the corresponding image from a display of four: the target word, two unrelated words, and a word from one of four contrast categories (i.e., voiced-initial [e.g., peach-beach], voiced-final [e.g., back-bag], manner-initial [e.g., talk-sock], and manner-final [e.g., bat-bass]). Fixations were monitored to measure how strongly words compete for recognition depending on the contrast type (voicing, manner) and location (word-initial or final). Manner contrasts competed more for recognition than did voicing contrasts, and contrasts that occurred in word-final position were harder to distinguish than word-initial position. These results are an important initial step toward creating an evidence-based hierarchy for auditory training for individuals who use cochlear implants.

Age and category structure in phonetic category learning

When learning new speech sound categories in a second language, listeners must decide which sounds belong to the same category and which belong to different categories. This learning process is contingent on perceptual systems that change across the lifespan, which may, in turn, make certain categories easier to learn depending on which systems are employed by learners. In the present study, learners across a variety of ages categorize speech sounds taken from German that vary in their category structure (i.e., the complexity of the categories being learned within the stimulus space). Participants aged 7 to 70 are recruited and run at community sites such as museums and libraries. The preliminary dataset indicates that the simpler category structure saw strong age effects (with poorer performance on the younger and older ends of the age range and better performance at intermediate ages), while differences across ages were much smaller for the more complex category structure. These findings suggest that learning simpler category structures may rely on age-dependent learning systems, while learning more complex categories may rely on systems that are less age-dependent.

Listeners' convergence towards an artificial agent in a joint phoneme categorization task

This study focuses on inter-individual convergence effects in the perception and categorization of speech sounds. We ask to what extent two listeners can come to establish a shared set of categorization criteria in a phoneme identification task that they accomplish together. Several hypotheses are laid out in the framework of a Bayesian model of speech perception that we have developed to account for how two listeners may each infer the parameters that govern their partner’s responses. In our experimental paradigm, participants were asked to perform a joint phoneme identification task with a partner that, unbeknownst to them, was an artificial agent, whose responses we manipulated along two dimensions, the location of the categorical boundary and the slope of the identification function. Convergence was found to arise for bias but not for slope. Numerical simulations suggested that lack of convergence in slope may stem from the listeners’ prior level of confidence in the variance in VOT for the two phonemic categories. This study sheds new light on perceptual convergence between listeners in the categorization of speech sounds, a phenomenon that has received little attention so far in spite of its central importance for speech communication.

Crossmodal associations between naturally occurring tactile and sound textures.

This study investigates the crossmodal associations between naturally occurring sound textures and tactile textures. Previous research has demonstrated the association between low-level sensory features of sound and touch, as well as higher-level, cognitively mediated associations involving language, emotions, and metaphors. However, stimuli like textures, which are found in both modalities have received less attention. In this study, we conducted two experiments: a free association task and a two alternate forced choice task using everyday tactile textures and sound textures selected from natural sound categories. The results revealed consistent crossmodal associations reported by participants between the textures of the two modalities. They tended to associate more sound textures (e.g., wood shavings and sandpaper) with tactile surfaces that were rated as harder, rougher, and intermediate on the sticky-slippery scale. While some participants based the auditory-tactile association on sensory features, others made the associations based on semantic relationships, co-occurrence in nature, and emotional mediation. Interestingly, the statistical features of the sound textures (mean, variance, kurtosis, power, autocorrelation, and correlation) did not show significant correlations with the crossmodal associations, indicating a higher-level association. This study provides insights into auditory-tactile associations by highlighting the role of sensory and emotional (or cognitive) factors in prompting these associations.

Movement-Related Modulation in Mouse Auditory Cortex Is Widespread Yet Locally Diverse.

Neurons in the mouse auditory cortex are strongly influenced by behavior, including both suppression and enhancement of sound-evoked responses during movement. The mouse auditory cortex comprises multiple fields with different roles in sound processing and distinct connectivity to movement-related centers of the brain. Here, we asked whether movement-related modulation in male mice might differ across auditory cortical fields, thereby contributing to the heterogeneity of movement-related modulation at the single-cell level. We used wide-field calcium imaging to identify distinct cortical fields and cellular-resolution two-photon calcium imaging to visualize the activity of layer 2/3 excitatory neurons within each field. We measured each neuron's responses to three sound categories (pure tones, chirps, and amplitude-modulated white noise) as mice rested and ran on a non-motorized treadmill. We found that individual neurons in each cortical field typically respond to just one sound category. Some neurons are only active during rest and others during locomotion, and those that are responsive across conditions retain their sound-category tuning. The effects of locomotion on sound-evoked responses vary at the single-cell level, with both suppression and enhancement of neural responses, and the net modulatory effect of locomotion is largely conserved across cortical fields. Movement-related modulation in auditory cortex also reflects more complex behavioral patterns, including instantaneous running speed and nonlocomotor movements such as grooming and postural adjustments, with similar patterns seen across all auditory cortical fields. Our findings underscore the complexity of movement-related modulation throughout the mouse auditory cortex and indicate that movement-related modulation is a widespread phenomenon.

Musical training is not associated with spectral context effects in instrument sound categorization.

Musicians display a variety of auditory perceptual benefits relative to people with little or no musical training; these benefits are collectively referred to as the "musician advantage." Importantly, musicians consistently outperform nonmusicians for tasks relating to pitch, but there are mixed reports as to musicians outperforming nonmusicians for timbre-related tasks. Due to their experience manipulating the timbre of their instrument or voice in performance, we hypothesized that musicians would be more sensitive to acoustic context effects stemming from the spectral changes in timbre across a musical context passage (played by a string quintet then filtered) and a target instrument sound (French horn or tenor saxophone; Experiment 1). Additionally, we investigated the role of a musician's primary instrument of instruction by recruiting French horn and tenor saxophone players to also complete this task (Experiment 2). Consistent with the musician advantage literature, musicians exhibited superior pitch discrimination to nonmusicians. Contrary to our main hypothesis, there was no difference between musicians and nonmusicians in how spectral context effects shaped instrument sound categorization. Thus, musicians may only outperform nonmusicians for some auditory skills relevant to music (e.g., pitch perception) but not others (e.g., timbre perception via spectral differences).