Time-frequency Representation Research Articles

Low-Rank Constrained Multi-Kernel General Parameterized Time-Frequency Transform and Its Application in Rotating Machine Fault Diagnosis Under Varying Speed Condition

Low-Rank Constrained Multi-Kernel General Parameterized Time-Frequency Transform and Its Application in Rotating Machine Fault Diagnosis Under Varying Speed Condition

Enhanced Micro-Doppler Feature Extraction Using Adaptive Short-Time Kernel-Based Sparse Time-Frequency Distribution

The extraction of the micro-Doppler (m-D) feature based on time-frequency distribution (TFD) is of great significance for target detection and identification. To improve the feature extraction performance, numerous TFDs have been developed, with the majority falling under Cohen’s class. Nevertheless, these TFDs basically face a trade-off between artifact suppression and energy concentration. The main reason is that each Cohen’s class TFD is constructed by applying the two-dimensional Fourier transform to a kerneled ambiguity function directly, while existing kernels generally attenuate artifacts at the expense of losing valuable information. In this paper, a TFD reconstruction method employing an adaptive short-time kernel (ASTK) is developed in the framework of sparse representation (SR) theory to overcome this trade-off and enhance the m-D feature. Firstly, the task of the optimal kernel is explained from the viewpoint of the instantaneous auto-correlation function (IAF). Secondly, based on the quasi-linear frequency modulation feature of most m-D signals during short-time periods, the distribution rule of the short-time IAF (STIAF) in the ambiguity plane is concluded. Guided by this rule, an ASTK that can effectively remove unwanted artifacts with the least information loss is designed. Finally, an SR-based reconstruction procedure is conducted on the kerneled STIAF to generate an artifact-free TFD with high energy concentration, which can effectively enhance the m-D feature. Experiments using both simulated and real-world m-D signals demonstrate the effectiveness of the proposed method.

Automatic Seizure Detection Based on Stockwell Transform and Transformer.

Epilepsy is a chronic neurological disease associated with abnormal neuronal activity in the brain. Seizure detection algorithms are essential in reducing the workload of medical staff reviewing electroencephalogram (EEG) records. In this work, we propose a novel automatic epileptic EEG detection method based on Stockwell transform and Transformer. First, the S-transform is applied to the original EEG segments, acquiring accurate time-frequency representations. Subsequently, the obtained time-frequency matrices are grouped into different EEG rhythm blocks and compressed as vectors in these EEG sub-bands. After that, these feature vectors are fed into the Transformer network for feature selection and classification. Moreover, a series of post-processing methods were introduced to enhance the efficiency of the system. When evaluating the public CHB-MIT database, the proposed algorithm achieved an accuracy of 96.15%, a sensitivity of 96.11%, a specificity of 96.38%, a precision of 96.33%, and an area under the curve (AUC) of 0.98 in segment-based experiments, along with a sensitivity of 96.57%, a false detection rate of 0.38/h, and a delay of 20.62 s in event-based experiments. These outstanding results demonstrate the feasibility of implementing this seizure detection method in future clinical applications.

A Scalogram-Based CNN Approach for Audio Classification in Construction Sites

The automatic monitoring of activities in construction sites through the proper use of acoustic signals is a recent field of research that is currently in continuous evolution. In particular, the use of techniques based on Convolutional Neural Networks (CNNs) working on the spectrogram of the signal or its mel-scale variants was demonstrated to be quite successful. Nevertheless, the spectrogram has some limitations, which are due to the intrinsic trade-off between temporal and spectral resolutions. In order to overcome these limitations, in this paper, we propose employing the scalogramas a proper time–frequency representation of the audio signal. The scalogram is defined as the square modulus of the Continuous Wavelet Transform (CWT) and is known as a powerful tool for analyzing real-world signals. Experimental results, obtained on real-world sounds recorded in construction sites, have demonstrated the effectiveness of the proposed approach, which is able to clearly outperform most state-of-the-art solutions.

Reduced-rank spectral mixtures Gaussian processes for probabilistic time–frequency representations

Deterministic time–frequency representations are commonly used in signal processing, particularly in audio processing. Whilst presenting many potential advantages, their probabilistic counterparts are not widely used, essentially because of the computational load and the lack of clear interpretability of the different underlying models. However, using state space models, they have been shown recently to be equivalent to Spectral Mixtures Gaussian processes (SM-GP). This pioneer work unlocks this problem and opens the path for the development of tractable and interpretable probabilistic time–frequency analysis. In this paper, we propose a relatively simple yet a significant improvement of that work in terms of computational complexity, flexibility and practical application. To do so, we use a recent approach for covariance approximation to develop an algorithm for faster inference of SM-GP, while opting for a frequency-domain approach to hyperparameter learning. We illustrate the practical potential of our method using voiced speech data. We first show that key speech features can be accurately learned from the data. Second, we show that our method can yield better performances in denoising.

Time–space–frequency feature Fusion for 3-channel motor imagery classification

Low-channel EEG devices are crucial for portable and entertainment applications. However, the low spatial resolution of EEG presents challenges in decoding low-channel motor imagery. This study introduces TSFF-Net, a novel network architecture that integrates time–space–frequency features, effectively compensating for the limitations of single-mode feature extraction networks based on time-series or time–frequency modalities. TSFF-Net comprises four main components: time–frequency representation, time–frequency feature extraction, time–space feature extraction, and feature fusion and classification. Time–frequency representation and feature extraction transform raw EEG signals into time–frequency spectrograms and extract relevant features. The time–space network processes time-series EEG trials as input and extracts temporal–spatial features. Feature fusion employs Maximum Mean Discrepancy (MMD) loss to constrain the distribution of time–frequency and time–space features in the Reproducing Kernel Hilbert Space, subsequently combining these features using a weighted fusion approach to obtain effective time–space–frequency features. Moreover, few studies have explored the decoding of three-channel motor imagery based on time–frequency spectrograms. This study proposes a shallow, lightweight decoding architecture (TSFF-img) based on time–frequency spectrograms and compares its classification performance in low-channel motor imagery with other methods using two publicly available datasets. Experimental results demonstrate that TSFF-Net not only compensates for the shortcomings of single-mode feature extraction networks in EEG decoding, but also outperforms other state-of-the-art methods. Overall, TSFF-Net offers considerable advantages in decoding low-channel motor imagery and provides valuable insights for algorithmically enhancing low-channel EEG decoding.

Semi-Supervised Domain Adaptation for Individual Identification from Electrocardiogram Signals

When acquiring electrocardiogram (ECG) signals, the placement of electrode patches is crucial for acquiring electrocardiographic signals. Constant displacement positions are essential for ensuring the consistency of the ECG signal when used for individual identification. However, achieving constant placement of ECG electrode patches in every trial for data acquisition is challenging. This is because different individuals may attach patches, and even when the same person attaches them, it may be difficult to specify the exact position. Therefore, gathering ECG data from various locations is necessary. However, this process requires a substantial amount of labor and time, owing to the requirement for multiple attempts. Nonetheless, persisting with these efforts enables the endurance of some ECG differences. To reduce labor and time, we propose a semi-supervised domain adaptation for individual identification using ECG signals. The method operates with a full set of original ECG signals and a small set of ECG signals from different placements to account for the differences between the signals in the generative adversarial network (CycleGAN). Specifically, to train the CycleGAN, the ECG signals were transformed into time–frequency representations, and the trained generator was used to generate ECG signals to expand the small set of ECG signals from different placements. Subsequently, both the original and generated signals were used to train the classifier for individual identification. This scenario can also be applied to the classification of ECG signals from different sensors. The PTB-ECG dataset was used for this experiment. We found that the proposed method showed higher accuracy than when only the original ECG signals were used for the training classifier.

Multi-master event waveform stacking microseismic location method based on time-frequency transformation

Waveform stacking is frequently employed in the analysis of microseismic data due to its adaptability, effectiveness, and noise immunity. However, the low signal-to-noise ratio and inaccurate velocity model have a considerable negative impact on the waveform stacking location method's performance. A time-frequency transform-based characteristic function and the addition of a multi-master event method to waveform stacking location are presented as solutions to the aforementioned issues. In this paper, non-smooth statistical features of characteristic function are extracted and enhanced using time-frequency transform. Based on the two-pair double-difference method, multi-master events are introduced and a reference criterion for selecting the master events is proposed. Experiments with simulated and actual microseismic data show that the conversion of time-domain signals into time-frequency signals by wavelet transform can enhance the peak capacity of the signal and reduce the background noise, enhancing the imaging resolution. The introduction of multi-master events not only reduces the sensitivity of the waveform stacking method to velocity error but also improves the clustering effect of the master event method.

A study on BSR diagnosis of power seat frame based on STRE-VK method

To establish an efficient method for the buzz-squeak-rattle (BSR) noise diagnosis of faulty power seat frame on the noisy production line, the S-transform ridgeline extraction & Vold-Kalman filtering (STRE-VK) method was proposed for the purpose of revealing the time–frequency characteristics of vibration signals of the power seat frame. The extraction of ridgelines from the time–frequency distribution was achieved through the utilization of adaptive generalized S-transform, and the separation of multiple harmonic components was accomplished by the Vold-Kalman filtering algorithm. The simulation analysis results demonstrated that the STRE-VK method performed better in terms of extracting ridgelines and segregating harmonic components. Then, the STRE-VK method was employed to the online BSR diagnosis of faulty power seat frames, and the prominence ratio of critical bands inspired by ECMA-74 was further calculated as a metric to identify different BSR noise types based on the separated vibration components. The results showed that the rapid online BSR diagnosis of power seat frame can be achieved accurately.

Deep Wavelet Convolutional Neural Networks for Multimodal Human Activity Recognition Using Wearable Inertial Sensors.

Recent advances in wearable systems have made inertial sensors, such as accelerometers and gyroscopes, compact, lightweight, multimodal, low-cost, and highly accurate. Wearable inertial sensor-based multimodal human activity recognition (HAR) methods utilize the rich sensing data from embedded multimodal sensors to infer human activities. However, existing HAR approaches either rely on domain knowledge or fail to address the time-frequency dependencies of multimodal sensor signals. In this paper, we propose a novel method called deep wavelet convolutional neural networks (DWCNN) designed to learn features from the time-frequency domain and improve accuracy for multimodal HAR. DWCNN introduces a framework that combines continuous wavelet transforms (CWT) with enhanced deep convolutional neural networks (DCNN) to capture the dependencies of sensing signals in the time-frequency domain, thereby enhancing the feature representation ability for multiple wearable inertial sensor-based HAR tasks. Within the CWT, we further propose an algorithm to estimate the wavelet scale parameter. This helps enhance the performance of CWT when computing the time-frequency representation of the input signals. The output of the CWT then serves as input for the proposed DCNN, which consists of residual blocks for extracting features from different modalities and attention blocks for fusing these features of multimodal signals. We conducted extensive experiments on five benchmark HAR datasets: WISDM, UCI-HAR, Heterogeneous, PAMAP2, and UniMiB SHAR. The experimental results demonstrate the superior performance of the proposed model over existing competitors.

Sparse time-frequency analysis of seismic data via convolutional neural network

Time-frequency (TF) analysis is commonly used to reveal the local properties of seismic signals, such as frequency and spectral contents varying with time/depth. Aiming to realize a highly localized TF representation of seismic signals, researchers treated the TF analysis as an inverse problem, and the regularization is adopted in the objective functions. Traditionally, the TF sparse inversion process is solved by the Lasso regression. It has been proven that the Lasso regression needs a large number of iterations to reach a high accurate solution for the convex problem. Recently, convolutional neural network (CNN) has been successfully used to solve the convex problem due to their high computational efficiency and strong nonlinear characterization ability. We use CNN to solve the sparse TF inversion, and our method is called STFA-CNN. The objective function in the neural network architecture consists of two portions. The first one is to minimize the difference between the local forward and backward Fourier transform of seismic signals. The second is minimizing the regularization ([Formula: see text] norm) of TF results. To demonstrate the effectiveness of our method, we apply it to synthetic and real seismic data. We further use the calculated TF spectra to compute the attenuation of seismic waveforms and apply the attenuation attribute to predict the hydrocarbons of a seismic survey acquired over the Ordos Basin, northwest of China.

Sharp detection of oscillation packets in rich time-frequency representations of neural signals.

Brain oscillations most often occur in bursts, called oscillation packets, which span a finite extent in time and frequency. Recent studies have shown that these packets portray a much more dynamic picture of synchronization and transient communication between sites than previously thought. To understand their nature and statistical properties, techniques are needed to objectively detect oscillation packets and to quantify their temporal and frequency extent, as well as their magnitude. There are various methods to detect bursts of oscillations. The simplest ones divide the signal into band limited sub-components, quantifying the strength of the resulting components. These methods cannot by themselves cope with broadband transients that look like genuine oscillations when restricted to a narrow band. The most successful detection methods rely on time-frequency representations, which can readily show broadband transients and harmonics. However, the performance of such methods is conditioned by the ability of the representation to localize packets simultaneously in time and frequency, and by the capabilities of packet detection techniques, whose current state of the art is limited to extraction of bounding boxes. Here, we focus on the second problem, introducing two detection methods that use concepts derived from clustering and topographic prominence. These methods are able to delineate the packets' precise contour in the time-frequency plane. We validate the new approaches using both synthetic and real data recorded in humans and animals and rely on a super-resolution time-frequency representation, namely the superlets, as input to the detection algorithms. In addition, we define robust tests for benchmarking and compare the new methods to previous techniques. Results indicate that the two methods we introduce shine in low signal-to-noise ratio conditions, where they only miss a fraction of packets undetected by previous methods. Finally, algorithms that delineate precisely the border of spectral features and their subcomponents offer far more valuable information than simple rectangular bounding boxes (time and frequency span) and can provide a solid foundation to investigate neural oscillations' dynamics.

Time-frequency enhanced characterization method based on asymmetric image reconstruction autoencoder

The vibration signals of mechanical equipment are subject to the influence of complex and variable working conditions, often exhibiting non-smooth and non-linear characteristics. The conventional time-frequency (TF) analysis (TFA) method, which relies on energy concentration, is susceptible to noise and impact, making it challenging to accurately extract fault characteristics. To overcome these limitations, this paper proposes an innovative approach. In this paper, we introduce an asymmetric image reconstruction autoencoder model, which is based on two well-known TFA methods, namely, short-time Fourier transform (STFT) and synchroextracting transform (SET), effectively reducing noise and improving the TF energy concentration process through learning the mapping relationship between STFT and SET. To address the clarity issue in the reconstructed TF images, the paper incorporates a channel attention mechanism known as SE Block into the encoding-decoding structure. Additionally, a skip connection structure is introduced to aid in restoring the structural details of the reconstructed TF images. Moreover, an improved weighted joint loss function is proposed to adaptively enhance various types of TF distribution features. This enhancement ensures that different characteristics of TF distribution are adequately addressed during the reconstruction process. The proposed method is put to the test using both simulated signals and experimental signals from gearbox rolling bearing faults. The results demonstrate that compared to traditional TFA and post-processing methods, the proposed model exhibits superior capabilities in enhancing the TF characterization of multi-source time-varying signals. Furthermore, it demonstrates remarkable robustness to noise and can accurately extract instantaneous frequency. These findings point to the promising potential of this method for mechanical fault identification and diagnosis applications.

Multiple-frequency attribute blending via adaptive uniform manifold approximation and projection and its application on hydrocarbon reservoir delineation

Multifrequency attribute blending is a highly effective tool for characterizing hydrocarbon reservoirs. It begins by extracting multifrequency attributes of seismic data based on time-frequency transformation. Subsequently, a blending algorithm is used to fuse the extracted multifrequency components, thereby obtaining the interpretation results of the interested reservoirs. The red-green-blue (RGB) algorithm is commonly used to fuse the multifrequency components. However, it should be noted that the RGB blending algorithm can only fuse three frequency components, i.e., the low-, middle-, and high-frequency components. Moreover, it can occasionally introduce ambiguities, making it difficult to interpret areas that appear white or yellow. To address these issues, we develop a workflow for multiple-frequency component analysis to delineate hydrocarbon reservoirs. First, we apply the generalized S-transform to obtain the multiple-frequency components of seismic data. Then, the correlation analysis is developed and implemented to select the sensitive frequency components. Finally, we use the uniform manifold approximation and projection, a nonlinear dimension reduction algorithm, to blend the extracted multiple-frequency components and obtain reservoir interpretation results. We apply the suggested workflow to synthetic data and a 3D field data volume to evaluate its effectiveness. Our mathematical analysis demonstrates that the suggested workflow can effectively fuse multiple-frequency components to accurately characterize hydrocarbon reservoirs.

Fault diagnosis of offshore wind turbines based on component separable synchroextracting transform

The fault diagnosis of wind turbines is crucial for wind power generation. However, the violent variation in wind power and directions of offshore wind turbines often results in high nonstationary vibration signals, which poses a challenge for effective fault recognition. Time-frequency analysis (TFA) is a common method to reveal the time-varying frequency components caused by faults. In this paper, we propose a component separable synchroextracting transform (CSSET) by exploring the unique modulation characteristics of wind turbine vibration signals. For the method, a fundamental frequency is firstly estimated by a time-frequency ridge optimization method, and the potential fault characteristic frequency (FCF) set is constructed according to the modulation characteristics of the wind turbine vibration signal. Next, the instantaneous amplitudes (IAs) of these components in the set are estimated by a time-varying bandpass filter. Then, the instantaneous frequency (IF) of the reconstructed single component is extracted by the synchronous extraction operator (SEO), and a high-precision time-frequency representation (TFR) is obtained. The proposed method takes full advantage of the physical modulation characteristics of vibration signals and the frequency components which are most relevant to fault information are only preserved. The effectiveness of the proposed method is verified by analyzing simulation and experimental signals.

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Abstract The paper describes the relations of speech signal representation in the layers of the convolutional neural network. Using activation maps determined by the Grad-CAM algorithm, energy distribution in the time–frequency space and their relationship with prosodic properties of the considered emotional utterances have been analysed. After preliminary experiments with the expressive speech classification task, we have selected the CQT-96 time–frequency representation. Also, we have used a custom CNN architecture with three convolutional layers in the main experimental phase of the study. Based on the performed analysis, we show the relationship between activation levels and changes in the voiced parts of the fundamental frequency trajectories. As a result, the relationships between the individual activation maps, energy distribution, and fundamental frequency trajectories for six emotional states were described. The results show that the convolutional neural network in the learning process uses similar fragments from time–frequency representation, which are also related to the prosodic properties of emotional speech utterances. We also analysed the relations of the obtained activation maps with time-domain envelopes. It allowed observing the importance of the speech signals energy in classifying individual emotional states. Finally, we compared the energy distribution of the CQT representation in relation to the regions’ energy overlapping with masks of individual emotional states. In the result, we obtained information on the variability of energy distributions in the selected signal representation speech for particular emotions.

Enhancing Micro-Doppler Classification Using Superlet-Based Time-Frequency Distribution

Enhancing Micro-Doppler Classification Using Superlet-Based Time-Frequency Distribution

Sound Event Detection utilizing Spectro-Temporal Receptive Field

Sound event detection (SED) is a task of recognizing the target sound events with their respective onset and offset time in a recorded audio clip. SED is closely related to auditory scene analysis, the process by which human auditory systems perceive sound into perceptually meaningful components. While human auditory system is trained to perform SED well enough, the way sound is transformed into neural response within human auditory system still remain as the subject of ongoing research. One of efforts to describe the relationship between sound and neural response is the spectro-temporal receptive field (STRF). STRF acts as a linear function between sound time-frequency representation and primary auditory cortex (A1) cell response, so that neural response can be predicted by convolution of time-frequency representation of sound and STRF. In addition, STRF is designed to capture spectro-temporal modulation in that A1 cells are much reactive to spectro-temporally modulated ripple. In this work, we utilized STRF as a kernel of the first layer in the convolutional neural network (CNN) to extract neural response from input audio clip to make SED model similar to human auditory system. Then, we constructed two-branched SED model named as Two Branch STRF-Net (TB-STRFNet) composed of STRF branch and basic branch. The STRF branch extracts spectro-temporal modulation information and the basic branch extracts detailed and complex time-frequency information. TB-STRFNet outperformed the baseline by 4.3% in terms of main metric in DCASE 2023 Task 4 subtask B.

Non-intrusive Speech Intelligibility Prediction Method for Reverberant Speech Using Neural Network-Based Frequency Segmentation and Masking Front-end

In conventional non-intrusive speech intelligibility estimation, reverberation is extracted from the time-frequency representation of the input by explicit filter bank processing or spectral masking. However, these filter banks and masking processes are not always optimal. We replaced these processes with convolutional neural networks using rectangular kernels restricted to the frequency direction and masking such as a self-attention mechanism. We believe that this will enable feature extraction that is optimal for intelligibility estimation and will enable its estimation with high accuracy that generalizes well to input under various conditions. We further applied this front-end CNN to a previously proposed prediction model using speech enhancement. As a result, the estimation accuracy was improved compared to conventional front-ends using fixed filter banks, and this prediction showed a correlation coefficient with the subjective evaluation of 0.84 compared to 0.80 with the fixed filter bank.

Wind speed prediction model based on DWT and Randon Forest

Wind energy is one of the fastest power generation technologies in the power generation industry and one of the most cost-effective methods of generating electrical power. For system reliability, improving highly appropriate wind speed forecasting methods is desirable. The wavelet transform is a powerful mathematical technique that converts an analyzed signal into a time-frequency representation. This technique helps forecast non-stationary time series. The objective is to evaluate the performance of the "DWT-Random Forest" model in predicting wind speed through a comparative analysis of performance metrics (MSE, RMSE, and MAPE) with similar studies. Our motivation is rooted in the pressing need to improve wind forecasting methods to optimize renewable energy generation. The method involved implementing the model, which presented performance metrics: MSE of 0.0099, RMSE of 0.0996, and MAPE of 0.0779. However, comparative analysis with previous studies reveals that our model demonstrates competitive performance. The main result of this study is the finding that the "DWT-Random Forest" model exhibits a respectable performance in predicting wind speed, although there is room for improvement.