Time-frequency Masking Research Articles

Adaptive Filtering for Multi-Track Audio Based on Time–Frequency Masking Detection

There is a growing need to facilitate the production of recorded music as independent musicians are now key in preserving the broader cultural roles of music. A critical component of the production of music is multitrack mixing, a time-consuming task aimed at, among other things, reducing spectral masking and enhancing clarity. Traditionally, this is achieved by skilled mixing engineers relying on their judgment. In this work, we present an adaptive filtering method based on a novel masking detection scheme capable of identifying masking contributions, including temporal interchangeability between the masker and maskee. This information is then systematically used to design and apply filters. We implement our methods on multitrack music to improve the quality of the raw mix.

Enhancing visual seismocardiography in noisy environments with adaptive bidirectional filtering for Cardiac Health Monitoring

BackgroundWearable sensors have revolutionized cardiac health monitoring, with Seismocardiography (SCG) at the forefront due to its non-invasive nature. However, the substantial motion artefacts have hindered the translation of SCG-based medical applications, primarily induced by walking. In contrast, our innovative technique, Adaptive Bidirectional Filtering (ABF), surpasses these challenges by refining SCG signals more effectively than any motion-induced noise. ABF leverages a noise-cancellation algorithm, operating on the benefits of the Redundant Multi-Scale Wavelet Decomposition (RMWD) and the bidirectional filtering framework, to achieve optimal signal quality.MethodologyThe ABF technique is a two-stage process that diminishes the artefacts emanating from motion. The first step by RMWD is the identification of the heart-associated signals and the isolating samples with those related frequencies. Subsequently, the adaptive bidirectional filter operates in two dimensions: it uses Time-Frequency masking that eliminates temporal noise while engaging in non-negative matrix Decomposition to ensure spatial correlation and dorsoventral vibration reduction jointly. The main component that is altered from the other filters is the recursive structure that changes to the motion-adapted filter, which uses vertical axis accelerometer data to differentiate better between accurate SCG signals and motion artefacts.OutcomeOur empirical tests demonstrate exceptional signal improvement with the application of our ABF approach. The accuracy in heart rate estimation reached an impressive r-squared value of 0.95 at − 20 dB SNR, significantly outperforming the baseline value, which ranged from 0.1 to 0.85. The effectiveness of the motion-artifact-reduction methodology is also notable at an SNR of − 22 dB. Consequently, ECG inputs are not required. This method can be seamlessly integrated into noisy environments, enhancing ECG filtering, automatic beat detection, and rhythm interpretation processes, even in highly variable conditions. The ABF method effectively filters out up to 97% of motion-related noise components within the SCG signal from implantable devices. This advancement is poised to become an integral part of routine patient monitoring.

Multi-rolling element faults diagnosis of rolling bearing based on time-frequency analysis and multi-curves extraction

Abstract In industrial production, rolling bearings are widely used as key mechanical components in all types of rotating machinery. Fault diagnosis is essential for predicting bearing damage in advance, avoiding sudden equipment downtime and reducing economic losses. However, rolling element fault diagnosis of rolling bearings continues to be a challenge, especially with multi-rolling element faults. In view of the characteristics of randomness, weakness, and coupling in the vibration signal generated by multi-rolling element faults in rolling bearings, a multi-rolling element fault detection method is proposed by combination time-frequency (TF) analysis (TFA) with multi-curves extraction methods. The pre-processing method combined autoregressive model with maximum correlated kurtosis deconvolution is employed to enhance the weak periodic fault impulses in the raw vibration signals of the rolling bearing. Then an improved dynamic path multi-curves extraction method is proposed to extract multiple TF curves from the TF spectrogram (TFS) constructed via short-time Fourier transform. According to the proposed classification criteria, the TF curves are classified as homologous faults. The TF masking (TFM) method is employed to keep TF information closely associated with the fault impulse. Finally, the fault signals are reconstructed sequentially based on the TFS processed by TFM, and precise identification of multi-rolling element faults is achieved by envelope analysis. Experimental results demonstrate the effectiveness of the proposed method in extracting the weak fault features of multi-rolling elements and accomplishing fault separation and diagnosis.

A novel bearing health sensing technique using smartphone

Bearing health sensing has drawn increasing attention as bearing plays a vulnerable yet essential role in modern industry. Nevertheless, conventional health sensing technologies involve dedicated sensor installation and high costs, which may not meet the monitoring and sensing demands. Given this, we present a health sensing technique to capture the fault information of bearing by leveraging an off-the-shelf smartphone. Specifically, we first exploit only a readily available smartphone to capture the audio signal of bearing. Given the captured audio signal, an adaptive time–frequency masking is proposed to separate the noise, and the fault impulse is then sharpened by the singular value decomposition (SVD). The extensive experimental results illustrate that our technique is able to enhance the bearing health sensing capability of smartphones under various real-world scenarios. Even in noisy mining field conditions, the framework still achieves a notable performance improvement rate of 169 %.

A Wavelet-Based Approach for Motion Artifact Reduction in Ambulatory Seismocardiography.

Wearable sensing has become a vital approach to cardiac health monitoring, and seismocardiography (SCG) is emerging as a promising technology in this field. However, the applicability of SCG is hindered by motion artifacts, including those encountered in practice of which the strongest source is walking. This holds back the translation of SCG to clinical settings. We therefore investigated techniques to enhance the quality of SCG signals in the presence of motion artifacts. To simulate ambulant recordings, we corrupted a clean SCG dataset with real-walking-vibrational noise. We decomposed the signal using several empirical-mode-decomposition methods and the maximum overlap discrete wavelet transform (MODWT). By combining MODWT, time-frequency masking, and nonnegative matrix factorization, we developed a novel algorithm which leveraged the vertical axis accelerometer to reduce walking vibrations in dorsoventral SCG. The accuracy and applicability of our method was verified using heart rate estimation. We used an interactive selection approach to improve estimation accuracy. The best decomposition method for reduction of motion artifact noise was the MODWT. Our algorithm improved heart rate estimation from 0.1 to 0.8 r-squared at -15 dB signal-to-noise ratio (SNR). Our method reduces motion artifacts in SCG signals up to a SNR of -19 dB without requiring any external assistance from electrocardiography (ECG). Such a standalone solution is directly applicable to the usage of SCG in daily life, as a content-rich replacement for other wearables in clinical settings, and other continuous monitoring scenarios. In applications with higher noise levels, ECG may be incorporated to further enhance SCG and extend its usable range. This work addresses the challenges posed by motion artifacts, enabling SCG to offer reliable cardiovascular insights in more difficult scenarios, and thereby facilitating wearable monitoring in daily life and the clinic.

Semi-supervised Learning for Automatic Modulation Recognition Using Haar Time–Frequency Mask and Positional–Spatial Attention

Automatic modulation recognition plays an important role in many military and civilian applications, including cognitive radio, spectrum sensing, signal surveillance, and interference identification. Due to the powerful ability of deep learning to extract hidden features and perform classification, it can extract highly separative features from massive signal samples. Considering the condition of limited training samples, we propose a semi-supervised learning framework based on Haar time–frequency (HTF) mask data augmentation and the positional–spatial attention (PSA) mechanism. Specifically, the HTF mask is designed to increase data diversity, and the PSA is designed to address the limited receptive field of the convolutional layer and enhance the feature extraction capability of the constructed network. Extensive experimental results obtained on the public RML2016.10a dataset show that the proposed semi-supervised framework utilizes 1% of the given labeled data and reaches a recognition accuracy of 92.09% under 6 dB signals.

Improved STFT analysis using time-frequency masking for chatter detection in the milling process

In the field of machining, the purposeful increase in cutting depth and rotation speed can potentially cause chatter vibration. Chatter lowers surface quality, tool and spindle life, and produces excessive noise, which limits the productivity of the machining process. In this paper, an improved Short-Time Fourier Transform (STFT) involving time–frequency masking and adaptive thresholding is proposed. Initially, hammering tests were undertaken to ascertain the inherent natural frequency of the cutting system. Subsequently, milling tests were conducted across varied conditions utilizing an accelerometer sensor and a microphone (capturing sound signals) to acquire the requisite experimental data. A 3-level Variational Mode Decomposition (VMD) was used for pre-processing due to its inherentcapabilities for noise reduction. The chatter threshold is established by averaging the total intrinsic mode function (IMF) energy levels that have been selected. To demonstrate the efficacy of the proposed method, root mean square (RMS) and skewness were calculated as chatter indicators. The improved STFT time–frequency representation (TFR) was compared to the conventional TFR. Notably, both indicators demonstrate effectiveness and chatter sensitivity, while the improved STFT offers greater time–frequency resolution than its conventional counterpart and can be successfully used for monitoring the milling process.

EnViTSA: Ensemble of Vision Transformer with SpecAugment for Acoustic Event Classification.

Recent successes in deep learning have inspired researchers to apply deep neural networks to Acoustic Event Classification (AEC). While deep learning methods can train effective AEC models, they are susceptible to overfitting due to the models' high complexity. In this paper, we introduce EnViTSA, an innovative approach that tackles key challenges in AEC. EnViTSA combines an ensemble of Vision Transformers with SpecAugment, a novel data augmentation technique, to significantly enhance AEC performance. Raw acoustic signals are transformed into Log Mel-spectrograms using Short-Time Fourier Transform, resulting in a fixed-size spectrogram representation. To address data scarcity and overfitting issues, we employ SpecAugment to generate additional training samples through time masking and frequency masking. The core of EnViTSA resides in its ensemble of pre-trained Vision Transformers, harnessing the unique strengths of the Vision Transformer architecture. This ensemble approach not only reduces inductive biases but also effectively mitigates overfitting. In this study, we evaluate the EnViTSA method on three benchmark datasets: ESC-10, ESC-50, and UrbanSound8K. The experimental results underscore the efficacy of our approach, achieving impressive accuracy scores of 93.50%, 85.85%, and 83.20% on ESC-10, ESC-50, and UrbanSound8K, respectively. EnViTSA represents a substantial advancement in AEC, demonstrating the potential of Vision Transformers and SpecAugment in the acoustic domain.

Multiple faults separation and identification of rolling bearings based on time-frequency spectrogram

Rolling bearings play a crucial role as components in rotating machinery across various industrial fields. Bearing faults can potentially lead to severe accidents in operating machines. Therefore, condition monitoring and fault diagnosis of rolling bearings are essential for preventing equipment failures. Multiple faults are a common occurrence resulting from the prolonged operation of rolling bearings, and numerous research efforts have been made to study multiple faults in different components of the bearing. However, diagnosing multiple faults in a single component of the rolling bearing still remains a highly challenging task. In this paper, a multiple faults separation and identification method based on time-frequency (TF) spectrogram (TFS) is proposed for vibration signals of rolling bearings. Firstly, the fast path optimization method is improved to match the TFS of original vibration signals in bearing faults generated by short-time Fourier transform. Then multiple TF curves are extracted from the TFS by the proposed multiple transient component curves extraction method based on the improved fast path optimization method. With the fault characteristic period, a classification criterion is introduced to separate TF curves. Secondly, a TF masking method is constructed to retain the TF information closely related to fault components of vibration signals. Finally, the novel TF representation can be obtained to develop signal reconstruction, and multiple faults can be detected based on envelope analysis separately. The experiments from rolling bearings with multiple faults on raceways are used to verify the effectiveness of the proposed methods.

Improving intelligibility of dysarthric speech in noise for listeners with hearing loss

Noise reduction strategies based on time-frequency masking have been shown to provide substantial improvements in the intelligibility of speech in noise for both listeners with normal and impaired hearing. Previous studies on this type of noise reduction have involved intact, well-articulated speech spoken by healthy talkers (i.e., not disordered speech). However, the prevalence of speech disorders such as dysarthria is substantial, particularly in the aging population, which is highly susceptible to the impacts of hearing loss. We recently demonstrated the considerable impact of background noise on dysarthric speech as well as the effectiveness of time-frequency masking to improve the intelligibility of this disordered speech in noise for listeners with normal hearing (Borrie et al., 2023). Here, we present data on the feasibility of time-frequency masking to increase the intelligibility of dysarthric speech for listeners with sensorineural hearing impairment. Preliminary results indicate that the Ideal Quantized Mask, a method of time-frequency masking, significantly improves percent words correct scores of dysarthric speech in noise for listeners with hearing loss. Results will be discussed in relation to the specific impact of background noise on dysarthric speech, and the relationship between sensorineural hearing loss and dysarthric speech.

Seismic time-frequency masking for suppression of seismic speckle noise

Seismic speckle noise is the primary factor causing severe reflection distortions caused by small-scale near-surface scattering. As in the case of speckle noise in optics and acoustics, deterministic velocity model-building techniques cannot recover these heterogeneities which are much smaller than a wavelength. Conventional processing techniques struggle to perform when multiplicative noise remains unsuppressed. Although local and global stacking mitigates the effects of speckle noise, it leads to a severe loss of higher frequencies reducing the vertical resolution of the seismic data. The foundation for attacking speckle noise is a recently proposed mathematical model that includes two concurrent random multiplicative noise types: type 1 defining residual statics and type 2 describing random frequency-dependent phase perturbations that mimic small-scale near-surface scattering. Using this model, we have developed seismic time-frequency masking to suppress speckle noise on prestack data. The time-dependent and non-surface-consistent nature of scattering noise dictates window-based approaches borrowed from local stacking techniques. Separate manipulation of phase and amplitude spectra is achieved through modified time-frequency masking inspired by speech processing. A beamformed data set from local stacking is used as a guide for designing phase and amplitude masks that are directly applied to raw data in the time-frequency domain. The phase mask allows the restoration of coherency by repairing phase distortions caused by near-surface scattering. The amplitude mask corrects power spectrum (PS) distortions caused by multiplicative and additive noises. An amplitude mask is implemented using a data-driven minimum statistic approach that estimates the noise PS in each local window. The minimum statistics approach is adapted from speech processing using beamformed data as an initial signal estimate and as a guide to designing an improved amplitude mask. Synthetic and field data examples suggest significant improvements in coherency and signal-to-noise ratio after the suppression of multiplicative noise, which makes the data processable by conventional techniques subsequently.

An Improved Prototype Network and Data Augmentation Algorithm for Few-Shot Structural Health Monitoring Using Guided Waves

The significance of implementing online structural health monitoring (SHM) for aerospace structures under harsh service environments cannot be overemphasized. Deep learning has demonstrated a promising and effective means to achieve accommodate such a need. However, as envisaged, the performance of deep learning-facilitated SHM heavily relies on the scale of training dataset, degrading the practicability of the approach. To address this, we propose an improved prototype network and data augmentation methods for few-shot SHM using guided waves. In the improved prototype network, the weighted Euclidean distance is used for damage classification. An attention module is established to predict the weight coefficients. The Davies–Bouldin Index (DBI) is used in the loss function to better separate the embedding vectors of different classes. Time masking and frequency masking are proposed for data augmentation of guided wave signals. As bolt joints are widely used in aerospace structures, the proposed approach is experimentally validated by quantifying the degree of bolt loosening in multibolt connection structures. The results are compared against those obtained from other classical few-shot learning (FSL) methods.

Sound Event Localization and Detection Using Imbalanced Real and Synthetic Data via Multi-Generator

This study proposes a sound event localization and detection (SELD) method using imbalanced real and synthetic data via a multi-generator. The proposed method is based on a residual convolutional neural network (RCNN) and a transformer encoder for real spatial sound scenes. SELD aims to classify the sound event, detect the onset and offset of the classified event, and estimate the direction of the sound event. In Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Task 3, SELD is performed with a few real spatial sound scene data and a relatively large number of synthetic data. When a model is trained using imbalanced data, it can proceed by focusing only on a larger number of data. Thus, a multi-generator that samples real and synthetic data at a specific rate in one batch is proposed to prevent this problem. We applied the data augmentation technique SpecAugment and used time-frequency masking to the dataset. Furthermore, we propose a neural network architecture to apply the RCNN and transformer encoder. Several models were trained with various structures and hyperparameters, and several ensemble models were obtained by "cherry-picking" specific models. Based on the experiment, the single model of the proposed method and the model applied with the ensemble exhibited improved performance compared with the baseline model.

The Application of Time-Frequency Masking To Improve Intelligibility of Dysarthric Speech in Background Noise.

Background noise reduces speech intelligibility. Time-frequency (T-F) masking is an established signal processing technique that improves intelligibility of neurotypical speech in background noise. Here, we investigated a novel application of T-F masking, assessing its potential to improve intelligibility of neurologically degraded speech in background noise. Listener participants (N = 422) completed an intelligibility task either in the laboratory or online, listening to and transcribing audio recordings of neurotypical (control) and neurologically degraded (dysarthria) speech under three different processing types: speech in quiet (quiet), speech mixed with cafeteria noise (noise), and speech mixed with cafeteria noise and then subsequently processed by an ideal quantized mask (IQM) to remove the noise. We observed significant reductions in intelligibility of dysarthric speech, even at highly favorable signal-to-noise ratios (+11 to +23 dB) that did not impact neurotypical speech. We also observed significant intelligibility improvements from speech in noise to IQM-processed speech for both control and dysarthric speech across a wide range of noise levels. Furthermore, the overall benefit of IQM processing for dysarthric speech was comparable with that of the control speech in background noise, as was the intelligibility data collected in the laboratory versus online. This study demonstrates proof of concept, validating the application of T-F masks to a neurologically degraded speech signal. Given that intelligibility challenges greatly impact communication, and thus the lives of people with dysarthria and their communication partners, the development of clinical tools to enhance intelligibility in this clinical population is critical.

Maximizing environmental sound recognition and speech intelligibility using time-frequency masking

Environmental sound recognition is an essential part of the human auditory experience that not only provides a sense of connection to one’s surroundings but also forecasts potential nearby safety hazards. Unfortunately, important environmental sounds can be rendered inaudible or otherwise unrecognizable by modern noise-reduction technology, leading to reduced environmental sound recognition. What is needed is a system that simultaneously provides listeners with access to audible, recognizable environmental sounds and intelligible speech. Many modern noise-reduction systems rely on some form of time-frequency masking, such as the ideal ratio mask. Restricting the output range of this mask by limiting the maximum allowable attenuation of any given time-frequency unit results in a mask that preserves environmental sounds to a certain extent while enhancing speech. In an experiment, subjects with hearing impairment and normal hearing listened to mixtures of sound + speech that had been processed by time-frequency masks with various levels of maximum attenuation, resulting in different amounts of environmental sound preservation. In a dual-task paradigm, environmental sound recognition and speech intelligibility were measured, and it was found that both types of listeners simultaneously attain high levels of performance on both tasks when the attenuation of time-frequency units is limited to 23 dB. [Work supported by NIH F32DC019314, NIH R01DC015521, and The OSU Graduate School.]

A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection

In this paper, we propose a novel four-stage data augmentation approach to ResNet-Conformer based acoustic modeling for sound event localization and detection (SELD). First, we explore two spatial augmentation techniques, namely audio channel swapping (ACS) and multi-channel simulation (MCS), to deal with data sparsity in SELD. ACS and MDS focus on augmenting the limited training data with expanding direction of arrival (DOA) representations such that the acoustic models trained with the augmented data are robust to localization variations of acoustic sources. Next, time-domain mixing (TDM) and time-frequency masking (TFM) are also investigated to deal with overlapping sound events and data diversity. Finally, ACS, MCS, TDM and TFM are combined in a step-by-step manner to form an effective four-stage data augmentation scheme. Tested on the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 data set, our proposed augmentation approach greatly improves the system performance, ranking our submitted system in the first place in the SELD task of the DCASE 2020 Challenge. Furthermore, we employ a ResNet-Conformer architecture to model both global and local context dependencies of an audio sequence and win the first place in the DCASE 2022 SELD evaluations.

Multi-Channel Bin-Wise Speech Separation Combining Time-Frequency Masking and Beamforming

Multi-Channel Bin-Wise Speech Separation Combining Time-Frequency Masking and Beamforming

A Single-Channel Speech Enhancement Algorithm Combined with Time-Frequency Mask

Aiming at the problem of noise overestimation when the classical single-channel speech enhancement method suppresses noise, single-channel speech enhancement method on the basis of the time-frequency mask is put forward to improve the quality of speech enhancement and separation. First, we estimate the noise and prior signal-to-noise ratio from the noisy speech, calculate the time-frequency mask and finally combine noisy speech synthesis to enhance the speech. Adaptive time-frequency mask combines IBM and IRM to avoid over-suppressing noise. In this paper, noisy speech is processed by an improved time-frequency mask combined with SNR. The algorithm combines the advantages of the single-channel algorithm and time-frequency mask. The experimental results indicate that this method significantly improves the signal-to-noise ratio, is robust, and is easy to use.

Suppressing reverberation in cochlear implant stimulus patterns using time-frequency masks based on phoneme groups

Cochlear implant (CI) users experience considerable difficulty in understanding speech in reverberant listening environments. This issue is commonly addressed with time-frequency masking, where a time-frequency decomposed reverberant signal is multiplied by a matrix of gain values to suppress reverberation. However, mask estimation is challenging in reverberant environments due to the large spectro-temporal variations in the speech signal. To overcome this variability, we previously developed a phoneme-based algorithm that selects a different mask estimation model based on the underlying phoneme. In the ideal case where knowledge of the phoneme was assumed, the phoneme-based approach provided larger benefits than a phoneme-independent approach when tested in normal-hearing listeners using an acoustic model of CI processing. The current work investigates the phoneme-based mask estimation algorithm in the real-time feasible case where the prediction from a phoneme classifier is used to select the phoneme-specific mask. To further ensure real-time feasibility, both the phoneme classifier and mask estimation algorithm use causal features extracted from within the CI processing framework. We conducted experiments in normal-hearing listeners using an acoustic model of CI processing, and the results showed that the phoneme-specific algorithm benefitted the majority of subjects. [Work supported by NIH through Grant No. R01DC014290-05.]

Cycle GAN-Based Audio Source Separation Using Time–Frequency Masking

Cycle GAN-Based Audio Source Separation Using Time–Frequency Masking