Noisy Audio Research Articles

Wiener Filter with Convolutional Neural Network for Noise Removal in API-Based AI Models

This research aims to develop a robust Application Program Interface (API)-Based Artificial Intelligence (AI) system for effective noise removal from audio signals, enhancing speech quality and intelligibility in noisy environments to be fed into different AI models to assess the applicant interview. The proposed methodology combines sophisticated signal processing techniques and noise reduction algorithms with AI models trained on clean voice data and noise patterns. To achieve this goal, we leverage two key components: the Wiener filter and a Convolutional Neural Network (CNN). The Wiener filter serves as the foundational noise reduction technique, exploiting statistical properties of the signal and the noise to suppress unwanted noise components effectively. Concurrently, CNN is integrated to classify the clean and noisy audio. In this research, the best optimizers selected, including Adam, SGD, RMSprop, Adagrad, and Adadelta are evaluated to identify the most suitable classification. The optimizers evaluated through cross-validation and hold-out validation in the same batch size (25) and epoch (25) were used. The study demonstrates that the Adam optimizer yields the best results. The epoch was optimized to 35, 75, 105, and 125 and epoch of 105 was selected with accuracy of 99.52%, Recall of 100%, F1-Score of 99.50%, and ROC_AUC of 99.99% for cross-validation while Accuracy of 98.79%, Recall of 99.21%, F1-Score of 98.81%, and ROC_AUC of 99.54% for hold-out validation, significantly improving AI model performance. Lastly, we ensured the batch size parameter was suitable for our model by tuning it with different settings (25, 50, 75, and 125) using the optimized optimizer and epoch. The batch size of 25 yielded the best accuracy. The modeled CNN also included kernel regularization L2 to avoid overfitting.

Restoring Speaking Lips from Occlusion for Audio-Visual Speech Recognition

Prior studies on audio-visual speech recognition typically assume the visibility of speaking lips, ignoring the fact that visual occlusion occurs in real-world videos, thus adversely affecting recognition performance. To address this issue, we propose a framework that restores occluded lips in a video by utilizing both the video itself and the corresponding noisy audio. Specifically, the framework aims to achieve these three tasks: detecting occluded frames, masking occluded areas, and reconstruction of masked regions. We tackle the first two issues by utilizing the Class Activation Map (CAM) obtained from occluded frame detection to facilitate the masking of occluded areas. Additionally, we introduce a novel synthesis-matching strategy for the reconstruction to ensure the compatibility of audio features with different levels of occlusion. Our framework is evaluated in terms of Word Error Rate (WER) on the original videos, the videos corrupted by concealed lips, and the videos restored using the framework with several existing state-of-the-art audio-visual speech recognition methods. Experimental results substantiate that our framework significantly mitigates performance degradation resulting from lip occlusion. Under -5dB noise conditions, AV-Hubert's WER increases from 10.62% to 13.87% due to lip occlusion, but rebounds to 11.87% in conjunction with the proposed framework. Furthermore, the framework also demonstrates its capacity to produce natural synthesized images in qualitative assessments.

Visual Hallucination Elevates Speech Recognition

Due to the detrimental impact of noise on the conventional audio speech recognition (ASR) task, audio-visual speech recognition~(AVSR) has been proposed by incorporating both audio and visual video signals. Although existing methods have demonstrated that the aligned visual input of lip movements can enhance the robustness of AVSR systems against noise, the paired videos are not always available during inference, leading to the problem of the missing visual modality, which restricts their practicality in real-world scenarios. To tackle this problem, we propose a Discrete Feature based Visual Generative Model (DFVGM) which exploits semantic correspondences between the audio and visual modalities during training, generating visual hallucinations in lieu of real videos during inference. To achieve that, the primary challenge is to generate the visual hallucination given the noisy audio while preserving semantic correspondences with the clean speech. To tackle this challenge, we start with training the audio encoder in the Audio-Only (AO) setting, which generates continuous semantic features closely associated with the linguistic information. Simultaneously, the visual encoder is trained in the Visual-Only (VO) setting, producing visual features that are phonetically related. Next, we employ K-means to discretize the continuous audio and visual feature spaces. The discretization step allows DFVGM to capture high-level semantic structures that are more resilient to noise and generate visual hallucinations with high quality. To evaluate the effectiveness and robustness of our approach, we conduct extensive experiments on two publicly available datasets. The results demonstrate that our method achieves a remarkable 53% relative reduction (30.5%->12.9%) in Word Error Rate (WER) on average compared to the current state-of-the-art Audio-Only (AO) baselines while maintaining comparable results (< 5% difference) under the Audio-Visual (AV) setting even without video as input.

AUDIO SPECTROGRAM TRANSFORMER (AST): ADVANTAGES OVER TRADITIONAL ALGORITHMS IN SPEECH-TO-TEXT (STT)

Automatic Speech Recognition (ASR) has seen significant advancements in recent years, largely due to the development of deep learning models. One of the most notable advancements is the Spectrogram Transformer, a variant of the Transformer architecture tailored for audio processing tasks. In this paper, we review the Spectrogram Transformer and compare it with other traditional ASR algorithms. We discuss its benefits, such as improved performance on noisy audio and better modeling of long-range dependencies. Additionally, we explore its applications in various domains, including voice assistants, transcription services, and audio indexing. Through experiments on benchmark datasets like LibriSpeech and the Speech Commands Dataset, we demonstrate the effectiveness of the Spectrogram Transformer in achieving state-of-the-art performance. Our findings suggest that the Spectrogram Transformer offers a promising direction for future advancements in ASR technology.

Development of acoustic denoising learning network for communication enhancement in construction sites

This paper presents a novel denoising approach based on deep learning and signal processing to improve communication efficiency. Construction activities take place when different trades come to the site for overlapped periods to perform their works, which may easily produce hazardous noise levels. The existence of noise affects workers' health issues, especially hearing and rhythm of the heart, and impacts communication efficiency between workers. The proposed approach employs signal processing technique to transform the noisy audio into image and utilize neural networks to extract noisy features and denoise the image. The denoised image is then converted to obtain the denoised audio. Experiments on reducing the side effect of several common noises in construction sites were conducted, compared with the performance of denoising using conventional wavelet transform. Standard objective measures, such as signal-to-noise ratio (SNR), and subjective measures, such as listening tests are used for evaluations. Our experimental results show that the proposed algorithm achieved significant improvements over the traditional method, as evidenced by the following quantitative results of median value: MSE of 0.002, RMSE of 0.049, SNR of 5.7 dB, PSNR of 25.8 dB, and SSR of 8.Results indicate that the proposed algorithm outperforms conventional denoising methods in terms of both objective and subjective evaluation metrics and have the potential to facilitate communication between site workers when facing different noise sources inevitably.

Self-supervised speech denoising using only noisy audio signals

In traditional speech denoising tasks, clean audio signals are often used as the training target, but absolutely clean signals are collected from expensive recording equipment or in studios with the strict environments. To overcome this drawback, we propose an end-to-end self-supervised speech denoising training scheme using only noisy audio signals, named Only-Noisy Training (ONT), without extra training conditions. The proposed ONT strategy constructs training pairs only from each single noisy audio, and it contains two modules: training audio pairs generated module and speech denoising module. The first module adopts a random audio sub-sampler on each noisy audio to generate training pairs. The sub-sampled pairs are then fed into a novel complex-valued speech denoising module. Experimental results show that the proposed method not only eliminates the high dependence on clean targets of traditional audio denoising tasks, but also achieves on-par or better performance than other training strategies. Availability—ONT is available at https://github.com/liqingchunnnn/Only-Noisy-Training

TTS - VLSP 2021: Development of Smartcall Vietnamese Text-to-Speech

Recent advances in deep learning facilitate the development of end-to-end Vietnamese text-to-speech (TTS) systems with high intelligibility and naturalness in the presence of a clean training corpus. Given a rich source of audio recording data on the Internet, TTS has excellent potential for growth if it can take advantage of this data source. However, the quality of these data is often not sufficient for training TTS systems, e.g., noisy audio. In this paper, we propose an approach that preprocesses noisy found data on the Internet and trains a high-quality TTS model on the processed data. The VLSP-provided training data was thoroughly preprocessed using 1) voice activity detection, 2) automatic speech recognition-based prosodic punctuation insertion, and 3) Spleeter, source separation tool, for separating voice from background music. Moreover, we utilize a state-of-the-art TTS system that takes advantage of the Conditional Variational Autoencoder with the Adversarial Learning model. Our experiment showed that the proposed TTS system trained on the preprocessed data achieved a good result on the provided noisy dataset.

A Novel Human-Vehicle Interaction Assistive Device for Arab Drivers Using Speech Recognition

About one-quarter of all car collisions in the United States are caused by distracted driving, and this ratio is expected to rise. As vehicles are equipped with more elaborate and complex technology, human-vehicle interaction via dashboard displays and controls will become more complex and distracting. Human-vehicle interaction via voice-based technology offers a less distracting alternative. In this study we aim to develop a voice-based car assistant, with a focus on Arabic language speech recognition. We prepare a new 4000-word domain-specific lexicon to comprehensively support driver-vehicle interactions, and we create corresponding text and speech corpora. Then we extract acoustic feature vectors and use various acoustic models to support speech recognition. The language model is created using an n-gram model. Then acoustic and language models, and the lexicon are combined to generate a decoding graph. The text corpus consists of 6110 elements, including words, phrases, and sentences. The speech corpus has more than 60000 recordings (almost 50 hours). For the decoding of noise-free audio, a Deep Neural Network + Hidden Markov Model provided 94.832% accuracy, a Subspace Gaussian Mixture Model + Hidden Markov Model provided 94.2% accuracy, and the best Gaussian Mixture Model + Hidden Markov Model provided 94.13% accuracy. For the decoding of noisy audio, a Deep Neural Network + Hidden Markov Model provided 93.316% accuracy, a Subspace Gaussian Mixture Model + Hidden Markov Model provided 92.62% accuracy, and the best Gaussian Mixture Model + Hidden Markov Model provided 91.82% accuracy. A usability study was conducted on the system with 10 participants. Almost all of the results of that study showed usability ratings of greater than 4.0 out of 5.0. These usability ratings indicate that the proposed system was seen by the participants as important, and useful for reducing driver distraction.

Application of Fusion of Various Spontaneous Speech Analytics Methods for Improving Far-Field Neural-Based Diarization

Recently developed methods in spontaneous speech analytics require the use of speaker separation based on audio data, referred to as diarization. It is applied to widespread use cases, such as meeting transcription based on recordings from distant microphones and the extraction of the target speaker’s voice profiles from noisy audio. However, speech recognition and analysis can be hindered by background and point-source noise, overlapping speech, and reverberation, which all affect diarization quality in conjunction with each other. To compensate for the impact of these factors, there are a variety of supportive speech analytics methods, such as quality assessments in terms of SNR and RT60 reverberation time metrics, overlapping speech detection, instant speaker number estimation, etc. The improvements in speaker verification methods have benefits in the area of speaker separation as well. This paper introduces several approaches aimed towards improving diarization system quality. The presented experimental results demonstrate the possibility of refining initial speaker labels from neural-based VAD data by means of fusion with labels from quality estimation models, overlapping speech detectors, and speaker number estimation models, which contain CNN and LSTM modules. Such fusing approaches allow us to significantly decrease DER values compared to standalone VAD methods. Cases of ideal VAD labeling are utilized to show the positive impact of ResNet-101 neural networks on diarization quality in comparison with basic x-vectors and ECAPA-TDNN architectures trained on 8 kHz data. Moreover, this paper highlights the advantage of spectral clustering over other clustering methods applied to diarization. The overall quality of diarization is improved at all stages of the pipeline, and the combination of various speech analytics methods makes a significant contribution to the improvement of diarization quality.

Mixture of Inference Networks for VAE-Based Audio-Visual Speech Enhancement

We address unsupervised audio-visual speech enhancement based on variational autoencoders (VAEs), where the prior distribution of clean speech spectrogram is simulated using an encoder-decoder architecture. At enhancement (test) time, the trained generative model (decoder) is combined with a noise model whose parameters need to be estimated. The initialization of the latent variables describing the generative process of the clean speech via the decoder, is crucial, as the overall inference problem is non-convex. This is usually done by using the output of the trained encoder given the noisy audio and clean visual data as input. Current audio-visual VAE models do not provide an effective initialization because the two modalities are tightly coupled (concatenated) in the associated architectures. To overcome this issue, we introduce the mixture of inference networks variational autoencoder (MIN-VAE). Two encoder networks input, respectively, audio and visual data, and the posterior of the latent variables is modeled as a mixture of two Gaussian distributions output from each encoder. The mixture variable is also latent, and therefore learning the optimal balance between the audio and visual encoders is unsupervised as well. By training a shared decoder, the overall network learns to adaptively fuse the two modalities. Moreover, at test time, the visual encoder, taking (clean) visual data, is used for initialization. A variational inference approach is derived to train the proposed model. Thanks to the novel inference procedure and the robust initialization, the MIN-VAE exhibits superior performance than the standard audio-only as well as audio-visual counterparts.

GRACE: Generating Summary Reports Automatically for Cognitive Assistance in Emergency Response

EMS (emergency medical service) plays an important role in saving lives in emergency and accident situations. When first responders, including EMS providers and firefighters, arrive at an incident, they communicate with the patients (if conscious), family members and other witnesses, other first responders, and the command center. The first responders utilize a microphone and headset to support these communications. After the incident, the first responders are required to document the incident by filling out a form. Today, this is performed manually. Manual documentation of patient summary report is time-consuming, tedious, and error-prone. We have addressed these form filling problems by transcribing the audio from the scene, identifying the relevant information from all the conversations, and automatically filling out the form. Informal survey of first responders indicate that this application would be exceedingly helpful to them. Results show that we can fill out a model summary report form with an F1 score as high as 94%, 78%, 96%, and 83% when the data is noise-free audio, noisy audio, noise-free textual narratives, and noisy textual narratives, respectively.

Robust Audio Content Classification Using Hybrid-Based SMD and Entropy-Based VAD.

A robust approach for the application of audio content classification (ACC) is proposed in this paper, especially in variable noise-level conditions. We know that speech, music, and background noise (also called silence) are usually mixed in the noisy audio signal. Based on the findings, we propose a hierarchical ACC approach consisting of three parts: voice activity detection (VAD), speech/music discrimination (SMD), and post-processing. First, entropy-based VAD is successfully used to segment input signal into noisy audio and noise even if variable-noise level is happening. The determinations of one-dimensional (1D)-subband energy information (1D-SEI) and 2D-textural image information (2D-TII) are then formed as a hybrid feature set. The hybrid-based SMD is achieved because the hybrid feature set is input into the classification of the support vector machine (SVM). Finally, a rule-based post-processing of segments is utilized to smoothly determine the output of the ACC system. The noisy audio is successfully classified into noise, speech, and music. Experimental results show that the hierarchical ACC system using hybrid feature-based SMD and entropy-based VAD is successfully evaluated against three available datasets and is comparable with existing methods even in a variable noise-level environment. In addition, our test results with the VAD scheme and hybrid features also shows that the proposed architecture increases the performance of audio content discrimination.

A Novel Real-Time, Lightweight Chaotic-Encryption Scheme for Next-Generation Audio-Visual Hearing Aids

Next-generation audio-visual (AV) hearing aids stand as a major enabler to realize more intelligible audio. However, high data rate, low latency, low computational complexity, and privacy are some of the major bottlenecks to the successful deployment of such advanced hearing aids. To address these challenges, we propose an integration of 5G Cloud-Radio Access Network (C-RAN), Internet of Things (IoT), and strong privacy algorithms to fully benefit from the possibilities these technologies have to offer. Existing audio-only hearing aids are known to perform poorly in noisy situations where overwhelming noise is present. Current devices make the signal more audible but remain deficient in restoring intelligibility. Thus, there is a need for hearing aids that can selectively amplify the attended talker or filter out acoustic clutter. The proposed 5G IoT-enabled AV hearing-aid framework transmits the encrypted compressed AV information and receives encrypted enhanced reconstructed speech in real time to address cybersecurity attacks such as location privacy and eavesdropping. For security implementation, a real-time lightweight AV encryption is proposed, based on a piece-wise linear chaotic map (PWLSM), Chebyshev map, and a secure hash and S-Box algorithm. For speech enhancement, the received secure AV (including lip-reading) information in the cloud is used to filter noisy audio using both deep learning and analytical acoustic modelling. To offload the computational complexity and real-time optimization issues, the framework runs deep learning and big data optimization processes in the background, on the cloud. The effectiveness and security of the proposed 5G-IoT-enabled AV hearing-aid framework are extensively evaluated using widely known security metrics. Our newly reported, deep learning-driven lip-reading approach for speech enhancement is evaluated under four different dynamic real-world scenarios (cafe, street, public transport, pedestrian area) using benchmark Grid and ChiME3 corpora. Comparative critical analysis in terms of both speech enhancement and AV encryption demonstrates the potential of the envisioned technology to deliver high-quality speech reconstruction and secure mobile AV hearing aid communication. We believe our proposed 5G IoT enabled AV hearing aid framework is an effective and feasible solution and represents a step change in the development of next-generation multimodal digital hearing aids. The ongoing and future work includes more extensive evaluation and comparison with benchmark lightweight encryption algorithms and hardware prototype implementation.

A Fuzzy Approach to Mute Sensitive Information in Noisy Audio Conversations

Audio conversation between a service seeking customer and an agent are common in a voice based call center (VbCC) and are often recorded either for audit purposes or to enable the VbCC to improve their efficiency. These audio recordings invariably containpersonal information of the customer, often spoken by the customer to confirm their identity to get personalized services from the agent. This private to a person (P2aP) information is the recordings is a serious concern from the GDPR perspective and can lead to identity the ftamong other things. In this paper, we propose a robust framework that enables us to reliably spot the P2aP information in the audio and automatically mute it. The main contribution of this paper is the proposal of a novelfuzzy criteria which by design allows for reduced false alarms and at the same time increases the accuracy of the muting process even when the speech to text conversion process is erroneous. Evaluation on real call center conversations demonstrates the reliability of the proposed approach.

Windowed Adaptive Filtering for Reducing Noise in Audio Signals during Transmission to Remote Locations

Audio signals are so important to mankind nowadays that they are now carried over transmission lines to remote ends for entertainment purposes or to send messages across and other purposes. Incidentally these audio signals can be contaminated either at the source or along this transmission lines by unwanted signals such as Additive White Gaussian Noise (AWGN), Random Noise, power line noise and other noise signals. To preserve the integrity of the audio signals at the receiving end, any noise contaminating the signal must be reduced to the barest minimum. In this paper a finite impulse response filter based on LMS algorithm is developed to remove AWGN from audio signals. During the adaptation process of the filter, every updated coefficient sequence is modified with Blackman-Harris window before being applied to the noisy audio. After design the optimum sampling parameters of the filter are determined to be sampling frequency of 8000Hz, filter order of 29 and step size of 0.006. An audio signal is generated by loading the Hallelujah song in the M-file of matlab into work space. An AWGN is generated with the matlab and added to the generated audio signal to form a noisy or contaminated audio signal. When the noisy signal is applied to the designed filter result shows that the noise in the signal drastically reduced to give a clean audio signal. Listening to the uncontaminated, contaminated and filtered audio signals and comparing them confirms the effectiveness of the adaptive filter in audio signal processing.

MISNA - A musical instrument segregation system from noisy audio with LPCC-S features and extreme learning

MISNA - A musical instrument segregation system from noisy audio with LPCC-S features and extreme learning

MISNA - A musical instrument segregation system from noisy audio with LPCC-S features and extreme learning

Technology has developed a lot over the last decades and has made a profound impact in almost every field. The field of Music Information Retrieval (MIR) has not been an exception to this as well, one of its most promising applications being Automatic Music Transcription (AMT). It is important to identify the active regions of various Instruments in a piece before transcription and the challenge elevates even more when the audio clips are contaminated with noise. MISNA (Musical Instrument Segregation from Noisy Clips) is a system proposed towards the identification of isolated Instruments from noisy clips which can aid towards AMT in noisy environments. The system works using statistical features (LPCC-S) derived from raw Linear Predictive Cepstral Coefficient values on very short clips of lengths 1 and 2 seconds. The system has been tested for various SNR scenarios and highest accuracies of 98.63% and 97.42% for Individual Instruments and Instrument Family identification has been obtained with the aid of Extreme Learning based classifier for a highest of 2626 clips.

BUILDING A NOISY AUDIO DATASET TO EVALUATE MACHINE LEARNING APPROACHES FOR AUTOMATIC SPEECH RECOGNITION SYSTEMS

Automatic speech recognition systems are part of people's daily lives, embedded in personal assistants and mobile phones, helping as a facilitator for human-machine interaction while allowing access to information in a practically intuitive way. Such systems are usually implemented using machine learning techniques, especially with deep neural networks. Even with its high performance in the task of transcribing text from speech, few works address the issue of its recognition in noisy environments and, usually, the datasets used do not contain noisy audio examples, while only mitigating this issue using data augmentation techniques. This work aims to present the process of building a dataset of noisy audios, in a specific case of degenerated audios due to interference, commonly present in radio transmissions. Additionally, we present initial results of a classifier that uses such data for evaluation, indicating the benefits of using this dataset in the recognizer's training process. Such recognizer achieves an average result of 0.4116 in terms of character error rate in the noisy set (SNR = 30).

Innovative Method for Unsupervised Voice Activity Detection and Classification of Audio Segments

An accurate and noise-robust voice activity detection (VAD) system can be widely used for emerging speech technologies in the fields of audio forensics, wireless communication, and speech recognition. However, in real-life application, the sufficient amount of data or human-annotated data to train such a system may not be available. Therefore, a supervised system for VAD cannot be used in such situations. In this paper, an unsupervised method for VAD is proposed to label the segments of speech-presence and speech-absence in an audio. To make the proposed method efficient and computationally fast, it is implemented by using long-term features that are computed by using the Katz algorithm of fractal dimension estimation. Two databases of different languages are used to evaluate the performance of the proposed method. The first is Texas Instruments Massachusetts Institute of Technology (TIMIT) database, and the second is the King Saud University (KSU) Arabic speech database. The language of TIMIT is English, while the language of the KSU speech database is Arabic. TIMIT is recorded in only one environment, whereas the KSU speech database is recorded in distinct environments using various recording systems that contain sound cards of different qualities and models. The evaluation of the proposed method suggested that it labels voiced and unvoiced segments reliably in both clean and noisy audio.

Using Paired Distances of Signal Peaks in Stereo Channels as Fingerprints for Copy Identification

This article proposes to use the relative distances between adjacent envelope peaks detected in stereo audio as fingerprints for copy identification. The matching algorithm used is the rough longest common subsequence (RLCS) algorithm. The experimental results show that the proposed approach has better identification accuracy than an MPEG-7 based scheme for distorted and noisy audio. When compared with other schemes, the proposed scheme uses fewer bits with comparable performance. The proposed fingerprints can also be used in conjunction with the MPEG-7 based scheme for lower computational burden.