Sound Recognition Research Articles

Classification of pulmonary sounds through deep learning for the diagnosis of interstitial lung diseases secondary to connective tissue diseases

Early diagnosis of interstitial lung diseases secondary to connective tissue diseases is critical for the treatment and survival of patients. The symptoms, like dry cough and dyspnea, appear late in the clinical history and are not specific, moreover, the current approach to confirm the diagnosis of interstitial lung disease is based on high resolution computer tomography. However, computer tomography involves x-ray exposure for patients and high costs for the Health System, therefore preventing its use for a massive screening campaign in elder people.In this work we investigate the use of deep learning techniques for the classification of pulmonary sounds acquired from patients affected by connective tissue diseases. The novelty of the work consists of a suitably developed pre-processing pipeline for de-noising and data augmentation. The proposed approach is combined with a clinical study where the ground truth is represented by high resolution computer tomography. Various convolutional neural networks have provided an overall accuracy as high as 91% in the classification of lung sounds and have led to an overwhelming diagnostic accuracy in the range 91%−93%. Modern high performance hardware for edge computing can easily support our algorithms. This solution paves the way for a vast screening campaign of interstitial lung diseases in elder people on the basis of a non-invasive and cheap thoracic auscultation.

Classification of Normal and Abnormal Heart Sounds Using Empirical Mode Decomposition and First Order Statistic

Analysis of heart sound signals for automatic segmentation and classification has revealed in recent decades that it has the potential to detect pathology accurately in clinical applications. Various audio signal processing techniques have been used to reduce the subjectivity of heart sound analysis. This study aims to classify normal and abnormal heart sound signals. The feature extraction process was optimized by EMD and calculated using five first-order statistical parameters: mean, variance, kurtosis, skewness, and entropy. The classification system is optimized with a mutual information algorithm to select traits that can significantly improve system performance. In addition, the selection of the optimal system configuration also includes the k-fold cross-validation and kNN methods with k values ​​and the proper distance type. Based on the test results, the highest accuracy of 98.2% was obtained when the value of k = 1 and the type of cosine distance on kNN with a five-fold cross-validation system evaluation model.

Exploring the impact of noise and degradations on heart sound classification models

The development of data-driven heart sound classification models has been an active area of research in recent years. To develop such data-driven models in the first place, heart sound signals need to be captured using a signal acquisition device. However, it is almost impossible to capture noise-free heart sound signals due to the presence of internal and external noises in most situations. Such noises and degradations in heart sound signals can potentially reduce the accuracy of data-driven classification models. Although different techniques have been proposed in the literature to address the noise issue, how and to what extent different noise and degradations in heart sound signals impact the accuracy of data-driven classification models remains unexplored. To answer this question, we produced a synthetic heart sound dataset including normal and abnormal heart sounds contaminated with a variety of noise and degradations. We used this dataset to investigate the impact of noise and degradation in heart sound recordings on the performance of different classification models. The results show different noises and degradations affect the performance of heart sound classification models to a different extent; some are more problematic for classification models, and others are less destructive. Comparing the findings of this study with the results of a survey we previously carried out with a group of clinicians shows noise and degradations that are more detrimental to classification models are also more disruptive to accurate auscultation. The findings of this study can be leveraged to develop targeted heart sound quality enhancement approaches — which adapt the type and aggressiveness of quality enhancement based on the characteristics of noise and degradation in heart sound signals.

Recognition of brief sounds in rapid serial auditory presentation.

Two experiments were conducted to test the role of participant factors (i.e., musical sophistication, working memory capacity) and stimulus factors (i.e., sound duration, timbre) on auditory recognition using a rapid serial auditory presentation paradigm. Participants listened to a rapid stream of very brief sounds ranging from 30 to 150 milliseconds and were tested on their ability to distinguish the presence from the absence of a target sound selected from various sound sources placed amongst the distracters. Experiment 1a established that brief exposure to stimuli (60 to 150 milliseconds) does not necessarily correspond to impaired recognition. In Experiment 1b we found evidence that 30 milliseconds of exposure to the stimuli significantly impairs recognition of single auditory targets, but the recognition for voice and sine tone targets impaired the least, suggesting that the lower limit required for successful recognition could be lower than 30 milliseconds for voice and sine tone targets. Critically, the effect of sound duration on recognition completely disappeared when differences in musical sophistication were controlled for. Participants' working memory capacities did not seem to predict their recognition performances. Our behavioral results extend the studies oriented to understand the processing of brief timbres under temporal constraint by suggesting that the musical sophistication may play a larger role than previously thought. These results can also provide a working hypothesis for future research, namely, that underlying neural mechanisms for the processing of various sound sources may have different temporal constraints.

Research on recognition method of broiler overlapping sounds based on random forest and confidence interval

In response to the problem of difficult recognition of overlapping sounds in previous broiler health monitoring studies, in this paper, the author proposed a recognition method of broiler overlapping sounds based on random forest and confidence interval. The author newly defined the confidence rate, confidence interval and recognition error, and combined Random-Forest-based classifier (abbr. RF classifier) to realize the recognition of overlapping sounds in a fixed-duration broiler sound signal. In the research phase, the PXI-1050 model audio collection system was used to collect a large number of broiler sound signals, whereby multiple datasets corresponding to different sound types and overlapping sounds were established, and corresponding multiple first confidence rates were calculated. First, by analyzing the value interval of the first confidence rate of four sound types, including the cough, the crow, the purr and the flapping wing, respectively, the common value interval of the single sound type was obtained as [0.8, 1]. Second, by analyzing the value interval of the first confidence rate of the six sound type combinations in the double overlapping sounds, the common value interval of the double overlapping sounds was obtained as [0.45, 0.6]. On this basis, the overlapping interval was determined as [0, 0.7), and the pure interval was determined as [0.7, 1]. Then, by analyzing the value interval of the first confidence rate of four sound type combinations in the triple overlapping sounds, the common value interval of the triple overlapping sounds was obtained as [0.35, 0.45]. Therefore, within the overlapping interval, [0,0.35) was determined as the value interval of quadruple overlapping sounds, [0.35,0.45) was determined as the value interval of triple overlapping sounds, and [0.45,0.7) was determined as the value interval of double overlapping sounds. Finally, general procedure steps of the broiler overlapping sound recognition method used in the test phase was given. Multiple test results show that, for the recognition of sound types and overlapping sounds in broiler sound signals, the average recognition accuracy was 100%. For the recognition of overlapping degrees, the average recognition accuracy was 97.22%. For the recognition of sound type combinations, the average recognition error was 2.27%. It fully proves the good generalization effect of the method proposed in this paper, as well as its high practicability and robustness in practical tests. It is an important supplement to the existing broiler health monitoring study. Meanwhile, it can be applied to the research on recognition and separation of overlapping sounds or signals in similar fields, thus it has a high reference value.

A Successful Crowdsourcing Approach for Bird Sound Classification

Automated recorders are increasingly used in remote sensing of wildlife, yet automated methods of processing the audio remains challenging. Identifying animal sounds with machine learning provides a solution, but optimizing the models requires annotated training data. Producing such data can require much manual effort, which could be alleviated by engaging masses to contribute to research and share the workload. Birdwatchers are experts on identifying bird vocalizations and form an ideal focal audience for a citizen science project aiming for the required multitudes of annotated avian audio data. For this purpose, we launched a web portal that was targeted and advertised to Finnish birdwatchers. The users were asked to complete two kinds of tasks: 1) classify if a given bird sound belonged to the focal species and 2) classify all the bird species vocalizing in 10-second audio clips. In less than a year, the portal achieved annotations for 244,300 bird sounds and 5,358 clips, and attracted, on average, 70 visitors on daily basis. More than 200 birdwatchers took part in the classification tasks, of which 17 and 4 most dedicated users produced over half of the sound and clip classifications, respectively. As expected of birder experts, the classifications among users were highly consistent (mean agreement scores between 0.85–0.95, depending on the audio type) and resulted in high-quality training data for parameterizing machine learning models. Feedback about the web portal suggested that additional functionality such as increased freedom of choice would increase user motivation and dedication.

Deep-Learning-Based Sound Classification Model for Concrete Pouring Work Monitoring at a Construction Site

In the present study, the utilization of sound data in research and technology is examined, data classification techniques are analyzed, and the applicability and necessity of these techniques are explored in order to propose an acoustic classification model that differentiates between normal and abnormal sounds during concrete pouring. The paper presents an experiment in which normal sound data occurring during concrete pouring, main noise data from construction, and symptom data that could affect structural quality or even cause a collapse incident were collected. By analyzing sound data from actual construction sites and experiments, a deep-learning-based classification model was developed with the aim of preventing events that could compromise the quality and safety of structures in advance. In the classification model, both CNN (convolutional neural network) and RNN (recurrent neural network) exhibited high accuracies of 94.38% and 93.26%, respectively, demonstrating remarkable performance in identifying the status of concrete placement. Unlike previous research that only collected and sorted normal construction-related sound data, the current study developed a sorting model that addresses quality- and safety-related matters by including sound data that may influence material separation, concrete leakage, and formwork collapse during concrete placement, and differentiating these sounds from normal concrete pouring sounds. The research findings are expected to contribute to the improvement of safety management and work efficiency at construction sites.

Simulation research on music feature recognition based on mobile big data and smart sensors

Abstract With the increasing importance of music information retrieval, the construction of effective music recognition methods has gradually become one of the research focuses. The vocal map features of the collected music pronunciation signals are extracted. The research in this paper is primarily based on the basic physical characteristics of music notes; once these characteristics are identified, then they are mathematically extracted and analysed (voiceprint feature method), and a recognition model is established. On this basis, the process of adaptive separation and recognition of music signals is completed. Finally, the performance of different recognition types is verified and evaluated through experiments. The results show that the average accuracy rate of the modified algorithm within a certain range reaches 73.6%; additionally, the average accuracy rate is increased by 10.95% compared with the audio recognition based on Internet of Things (IoT) data, and is more accurate than the audio recognition method based on data collection, showing an improvement of 20.75%. This shows that the modified recognition algorithm adopted in the present research has stable and high accuracy for comprehensive music types with different characteristics. Finally, the identification method proposed in this paper shortens the time by 85.71% compared with the identification method of data collection, and shortens the identification time by 83.33% compared with the IoT identification method. This greatly improves the recognition of different musical feature types.

Auditory environment diversity quantified using entropy from real-world hearing aid data.

Using data collected from hearing aid users' own hearing aids could improve the customization of hearing aid processing for different users based on the auditory environments they encounter in daily life. Prior studies characterizing hearing aid users' auditory environments have focused on mean sound pressure levels and proportions of environments based on classifications. In this study, we extend these approaches by introducing entropy to quantify the diversity of auditory environments hearing aid users encounter. Participants from 4 groups (younger listeners with normal hearing and older listeners with hearing loss from an urban or rural area) wore research hearing aids and completed ecological momentary assessments on a smartphone for 1 week. The smartphone was programmed to sample the processing state (input sound pressure level and environment classification) of the hearing aids every 10 min and deliver an ecological momentary assessment every 40 min. Entropy values for sound pressure levels, environment classifications, and ecological momentary assessment responses were calculated for each participant to quantify the diversity of auditory environments encountered over the course of the week. Entropy values between groups were compared. Group differences in entropy were compared to prior work reporting differences in mean sound pressure levels and proportions of environment classifications. Group differences in entropy measured objectively from the hearing aid data were also compared to differences in entropy measured from the self-report ecological momentary assessment data. Auditory environment diversity, quantified using entropy from the hearing aid data, was significantly higher for younger listeners than older listeners. Entropy measured using ecological momentary assessment was also significantly higher for younger listeners than older listeners. Using entropy, we show that younger listeners experience a greater diversity of auditory environments than older listeners. Alignment of group entropy differences with differences in sound pressure levels and hearing aid feature activation previously reported, along with alignment with ecological momentary response entropy, suggests that entropy is a valid and useful metric. We conclude that entropy is a simple and intuitive way to measure auditory environment diversity using hearing aid data.

Research on Robust Audio-Visual Speech Recognition Algorithms

Automatic speech recognition (ASR) that relies on audio input suffers from significant degradation in noisy conditions and is particularly vulnerable to speech interference. However, video recordings of speech capture both visual and audio signals, providing a potent source of information for training speech models. Audiovisual speech recognition (AVSR) systems enhance the robustness of ASR by incorporating visual information from lip movements and associated sound production in addition to the auditory input. There are many audiovisual speech recognition models and systems for speech transcription, but most of them have been tested based in a single experimental setting and with a limited dataset. However, a good model should be applicable to any scenario. Our main contributions are: (i) Reproducing the three best-performing audiovisual speech recognition models in the current AVSR research area using the most famous audiovisual databases, LSR2 (Lip Reading Sentences 2) LSR3 (Lip Reading Sentences 3), and comparing and analyzing their performances under various noise conditions. (ii) Based on our experimental and research experiences, we analyzed the problems currently encountered in the AVSR domain, which are summarized as the feature-extraction problem and the domain-generalization problem. (iii) According to the experimental results, the Moco (momentum contrast) + word2vec (word to vector) model has the best AVSR effect on the LRS datasets regardless of whether there is noise or not. Additionally, the model also produced the best experimental results in the experiments of audio recognition and video recognition. Our research lays the foundation for further improving the performance of AVSR models.

A Genetic Algorithm Approach to Automate Architecture Design for Acoustic Scene Classification

Convolutional neural networks (CNNs) have been widely used with remarkable success in the acoustic scene classification (ASC) task. However, the performance of these CNNs highly relies on their architectures, requiring a lot of effort and expertise to design CNNs suitable for the investigated problem. In this work, we propose an efficient genetic algorithm (GA) that aims to find optimized CNN architectures for the ASC task. The proposed algorithm uses frequency-dimension splitting of the input spectrograms in order to explore the architecture search space in sub-CNN models in addition to classical single-path CNNs. Specifically, this algorithm aims to find the best number of sub-CNNs in addition to their architectures to better capture the distinct features of the input spectrograms. The proposed GA is specifically designed for sound classification to suit the ASC task than many other GAs that optimize conventional single-path CNN architectures. Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed method. Specifically, the proposed algorithm has achieved around 17.8%, 16%, and 17.2%, relative improvement in accuracy with respect to the baseline systems on the development datasets of DCASE2018-Task1A, DCASE2019-Task1A, and DCASE2020-Task1A, respectively.

Convolution neural networks with hybrid feature extraction methods for classification of voice sound signals

The convolutional neural networks (CNNs) lead in the domain of Sound Recognition due to its flexibility and ability with different adjusting parameters. The recognition of spoken English Alphabets by different people with deep learning techniques attracted the research community. In this paper, we are exploring the use of convolutional neural network (CNN), a deep learner that can automatically learn features directly from the dataset while training for the classification of sounds signals of English alphabets. In this proposed work, we consider two CNN architectures. In first architecture, we propose MFCC based features for pretrained two convolutional layer CNN architecture. In the second architecture, we propose a hybrid feature extraction method to train a block-based CNN architecture. The proposed systems consist of two components namely hybrid feature extraction and CNN classifier. The five auditory features log-Mel spectrogram (LM), MFCC, chroma, spectral contrast and Tonnetz features are extracted and then LM & MFCC are combined as one feature set. LM, MFCC, and CST features are aggregated as another for training to the proposed two CNNs, respectively. The different sound samples of English alphabets are collected from different people of different age groups. The feature sets collected from the hybrid feature extraction methods are presented to both the proposed CNNs and the experimental results are collected. The experimental results indicate that the taxonomic accuracy of the proposed architectures can surpass the existing methods of CNNs with single feature extraction methods. The proposed second architecture performs more effectively over the proposed first CNN architecture.

A lung sound recognition model to diagnoses the respiratory diseases by using transfer learning.

Respiratory disease is one of the leading causes of death in the world. Through advances in Artificial Intelligence, it appears possible for the days of misdiagnosis and treatment of respiratory disease symptoms rather than their root cause to move behind us. The traditional convolutional neural network cannot extract the temporal features of lung sounds. To solve the problem, a lung sounds recognition algorithm based on VGGish- stacked BiGRU is proposed which combines the VGGish network with the stacked bidirectional gated recurrent unit neural network. A lung Sound Recognition Algorithm Based on VGGish-Stacked BiGRU is used as a feature extractor which is a pre-trained model used for transfer learning. The target model is built with the same structure as the source model which is the VGGish model and parameter transfer is done from the source model to the target model. The multi-layer BiGRU stack is used to enhance the feature value and retain the model. While fine-tuning of the parameter of VGGish is frozen which successfully improves the model. The experimental results show that the proposed algorithm improves the recognition accuracy of lung sounds and the recognition accuracy of respiratory diseases.

Efficient classification of the adventitious sounds of the lung through a combination of SVM-LSTM-Bayesian optimization algorithm with features based on wavelet bi-phase and bi-spectrum

Efficient classification of the adventitious sounds of the lung through a combination of SVM-LSTM-Bayesian optimization algorithm with features based on wavelet bi-phase and bi-spectrum

Kết hợp học sâu với SVM để nâng cao chất lượng xử lý tiếng nói con người trong các thiết bị IoT

Along with the development of science and technology, especially internet of things (IOT), IOT-related products increasingly contribute to improving people’s life. Among those products, it is impossible not to mention smarts city, internet of Vehicle devices and especially smart home, which are usually voice controlled. Therefore, voice processing technology is also in need of improvement. The article mainly focuses on processing human voice independently of text. In particular, Convolutional network (CNN) and Support Vector Machine (SVM) will be integrated to create Feature Building Machine. SVMs are often used in voice and image classification, which accordingly is a critical and swift data sorter. The article analyzes the advantages of the combination Deep Neural Network (DNN) and SVMs in voice recognition and is the foundation to develop devices for smart home. The experimental results, which was used in the standard Voxceleb database, demonstrate the superiority in sound recognition compared to traditional i-vector methods or other CNN methods.

CNN and Bidirectional GRU-Based Heartbeat Sound Classification Architecture for Elderly People

Cardiovascular diseases (CVDs) are a significant cause of death worldwide. CVDs can be prevented by diagnosing heartbeat sounds and other conventional techniques early to reduce the harmful effects caused by CVDs. However, it is still challenging to segment, extract features, and predict heartbeat sounds in elderly people. The inception of deep learning (DL) algorithms has helped detect various types of heartbeat sounds at an early stage. Motivated by this, we proposed an intelligent architecture categorizing heartbeat into normal and murmurs for elderly people. We have used a standard heartbeat dataset with heartbeat class labels, i.e., normal and murmur. Furthermore, it is augmented and preprocessed by normalization and standardization to significantly reduce computational power and time. The proposed convolutional neural network and bi-directional gated recurrent unit (CNN + BiGRU) attention-based architecture for the classification of heartbeat sound achieves an accuracy of 90% compared to the baseline approaches. Hence, the proposed novel CNN + BiGRU attention-based architecture is superior to other DL models for heartbeat sound classification.

A late fusion deep neural network for robust speaker identification using raw waveforms and gammatone cepstral coefficients

Speaker identification aims at determining the speaker identity by analyzing his voice characteristics, and relies typically on statistical models or machine learning techniques. Frequency-domain features are by far the most used choice to encode the audio input in sound recognition. Recently, some studies have also analyzed the use of time-domain raw waveform (RW) with deep neural network (DNN) architectures. In this paper, we hypothesize that both time-domain and frequency-domain features can be used to increase the robustness of speaker identification task in adverse noisy and reverberation conditions, and we present a method based on a late fusion DNN using RWs and gammatone cepstral coefficients (GTCCs). We analyze the characteristics of RW and spectrum-based short-time features, reporting advantages and limitations, and we show that the joint use can increase the identification accuracy. The proposed late fusion DNN model consists of two independent DNN branches made primarily by convolutional neural networks (CNN) and fully connected neural networks (NN) layers. The two DNN branches have as input short-time RW audio fragments and GTCCs, respectively. The late fusion is computed on the predicted scores of the DNN branches. Since the method is based on short segments, it has the advantage of being independent from the size of the input audio signal, and the identification task can be computed by summing the predicted scores over several short-time frames. Analysis of speaker identification performance computed with simulations show that the late fusion DNN model improves the accuracy rate in adverse noise and reverberation conditions in comparison to the RW, the GTCC, and the mel-frequency cepstral coefficients (MFCCs) features. Experiments with real-world speech datasets confirm the efficiency of the proposed method, especially with small-size audio samples.

An integrated system for the acoustic monitoring of goat farms

Effective precision livestock farming requires the availability of reliable and up-to-date data characterizing the animals and environment of interest. This work presents the design, architecture, and implementation of a wireless acoustic sensor network for monitoring goat farms on a 24/7 basis. In addition, we define a hierarchical organization of the involved sound classes exhaustively covering every aspect of the encountered goat vocalizations. Moreover, we developed an annotation tool tailored to the specifics of the problem at hand, i.e. a big, real-world data environment, able to meaningfully assist the annotation of goat vocalizations via a suitable sound classification module. On top of that, a mobile phone application was developed facilitating authorized users to remotely access information describing the situation on the farm site. Importantly, such a non-invasive monitoring framework was installed in 4 different sites located in Northern Italy while taking into account their diverse characteristics.

A vector classifier for sound similarity classification based on audio features

This study introduces a vector classifier for Sound Similarity Classification. Sound classification using the known features can be accepted as the prime definition within a limited time. However, the sound is a function not only of spectral composition but also a change of this composition in time Then, it should be characterized with flexible classifier vectors as a function of time segment. The study considers (1) steady-state time segmentation of sound with respect to feature deviations and creating time-marked classifier vectors within the segment made of the know features and (2) using long-time combined segments and histogram base classifiers within the classifier vector. The main subject of the work is the reduction of memory and computation time effectiveness and similarity detection capability.

The causal properties of sounds predict their recognition and misidentification

One way to study human sound recognition is to investigate the reasons why sounds are sometimes misheard as coming from the wrong source. Understanding this cognitive process can not only help prevent undesirable sound confusions (e.g., auditory display design) but can also promote useful confusions (e.g., Foley effects, cognitive reappraisal for misophonia). We tested the hypothesis that sounds are more confusable if their source events share causal properties. In Expi.1, listeners assessed causal properties of everyday sounds (ESC-50 Dataset) by judging their actions (e.g., tapping), materials (e.g., metal), and causal agents (e.g., machines). Causal similarity between sounds was measured by the distance between their causal properties. In Exp. 2, new listeners identified these sounds with 90% accuracy. Using the distances obtained in Exp. 1, misidentifications were predicted with 91% sensitivity and 89% specificity. The causal properties that had the largest effect on recognition accuracy were predominantly actions (according to a Ridge regression). Additional experiments with our own recorded and synthesized sounds show that spectral degradation and temporal manipulation can cause significant changes in causal properties and source assignment. Altogether, these findings show that causal properties provide unique explanatory power for sound recognition and misidentification. [Work supported by REAM.]