Raw Waveform Research Articles

Phantom in the opera: adversarial music attack for robot dialogue system

This study explores the vulnerability of robot dialogue systems' automatic speech recognition (ASR) module to adversarial music attacks. Specifically, we explore music as a natural camouflage for such attacks. We propose a novel method to hide ghost speech commands in a music clip by slightly perturbing its raw waveform. We apply our attack on an industry-popular ASR model, namely the time-delay neural network (TDNN), widely used for speech and speaker recognition. Our experiment demonstrates that adversarial music crafted by our attack can easily mislead industry-level TDNN models into picking up ghost commands with high success rates. However, it sounds no different from the original music to the human ear. This reveals a serious threat by adversarial music to robot dialogue systems, calling for effective defenses against such stealthy attacks.

An explainable deepfake of speech detection method with spectrograms and waveforms

Research on deepfake techniques for speech is crucial for combatting the spread of fake information, safeguarding public privacy, and advancing forensic techniques. However, the lack of transparency and explainability of spoofed speech detection models raises concerns about their reliability. In this paper, we suggest using raw waveform signals and spectrograms as fused features of the spoofed speech detection model. We use the SHAP method to analyze the feature distribution of spoofed speech detection and explain the likelihood of fake speech. Our experimental results demonstrate that our approach achieves better classification results with lighter model parameters than other feature fusion methods. Finally, the feature contribution values are calculated under the SHAP method to visualize them as heat maps. It helps researchers to analyze the feature distribution of spoofed speech to identify the most critical features that distinguish between spoofed and bona fide and to ensure transparency in their use.

An acoustic emission based approach for damage pattern recognition in composite using linear discriminant analysis

With the increasing application of composite components in various industries, the assessment of their structural integrity, the analysis of damage processes, and the identification of damage patterns are becoming increasingly important. The accuracy of the analysis relies heavily on the selection of features. This paper presents a new idea to extract effective damage features from acoustic emission (AE) signals and accurately identify different damages in the failure process of carbon fiber reinforced polymer specimens. The method combines Hilbert–Huang transform (HHT) and Linear Discriminant Analysis (LDA) to analyze the AE signals generated during the damage process of composite specimens. Specifically, the Hilbert marginal energy spectrum of the signals was regarded as frequency domain descriptors. The frequency domain descriptors were subsequently associated with the parametric features after dimensionality reduction by LDA to construct the classification framework named HHT-LDA. The results show that the frequency domain descriptors of the AE signals associated with each damage mode characterized distinctly. The frequency band of energy distribution in the raw waveform for matrix cracking, delamination, and fiber breakage are (100–150 kHz), (150–300 kHz), and (300–350 kHz), respectively. In addition, the three damage patterns mentioned above were successfully detected and recognized from the complex AE waveforms using HHT-LDA with 85% overall classification rate. This research idea will serve as a potential method for future composite damage pattern recognition and provide supporting knowledge for practical applications of AE monitoring.

Predicting Individual Survival Distributions Using ECG: A Deep Learning Approach Utilizing Features Extracted by a Learned Diagnostic Model

In the field of healthcare, individual survival prediction is important for personalized treatment planning. This study presents machine learning algorithms for predicting Individual Survival Distributions (ISD) using electrocardiography (ECG) data in two different formats. The models, which predict time until death, are developed and evaluated on a large, population-based cohort from Alberta, Canada. Our results demonstrate that models trained on raw ECG waveforms significantly outperform those trained on traditional ECG measurements in several metrics, including concordance index, hinge L1 loss, margin L1 loss, and margin truncated L1 loss. Additionally, the integration of predicted probabilities from wide-range diagnostic tasks not only enhances our ISD models' performance but also makes them significantly superior to other models across all evaluation metrics in individual survival prediction tasks. This innovative approach highlights the potential to leverage insights from diagnostic models for prognostic tasks, such as individual survival prediction. These findings could have far-reaching implications for the development of personalized treatment plans and open new avenues for future research in survival prediction using ECGs.

Variational Autoencoders for Biomedical Signal Morphology Clustering and Noise Detection.

Accurate estimation of physiological biomarkers using raw waveform data from non-invasive wearable devices requires extensive data preprocessing. An automatic noise detection method in time-series data would offer significant utility for various domains. As data labeling is onerous, having a minimally supervised abnormality detection method for input data, as well as an estimation of the severity of the signal corruptness, is essential. We propose a model-free, time-series biomedical waveform noise detection framework using a Variational Autoencoder coupled with Gaussian Mixture Models, which can detect a range of waveform abnormalities without annotation, providing a confidence metric for each segment. Our technique operates on biomedical signals that exhibit periodicity of heart activities. This framework can be applied to any machine learning or deep learning model as an initial signal validator component. Moreover, the confidence score generated by the proposed framework can be incorporated into different models' optimization to construct confidence-aware modeling. We conduct experiments using dynamic time warping (DTW) distance of segments to validated cardiac cycle morphology. The result confirms that our approach removes noisy cardiac cycles and the remaining signals, classified as clean, exhibit a 59.92% reduction in the standard deviation of DTW distances. Using a dataset of bio-impedance data of 97885 cardiac cycles, we further demonstrate a significant improvement in the downstream task of cuffless blood pressure estimation, with an average reduction of 2.67 mmHg root mean square error (RMSE) of Diastolic Blood pressure and 2.13 mmHg RMSE of systolic blood pressure, with increases of average Pearson correlation of 0.28 and 0.08, with a statistically significant improvement of signal-to-noise ratio respectively in the presence of different synthetic noise sources. This enables burden-free validation of wearable sensor data for downstream biomedical applications.

Robust voice activity detection using an auditory-inspired masked modulation encoder based convolutional attention network

Deep learning has revolutionized voice activity detection (VAD) by offering promising solutions. However, directly applying traditional features, such as raw waveforms and Mel-frequency cepstral coefficients, to deep neural networks often leads to degraded VAD performance due to noise interference. In contrast, humans possess the remarkable ability to discern speech in complex and noisy environments, which motivated us to draw inspiration from the human auditory system. We propose a robust VAD algorithm called auditory-inspired masked modulation encoder based convolutional attention network (AMME-CANet) that integrates our AMME with CANet. Firstly, we investigate the design of auditory-inspired modulation features as a deep-learning encoder (AME), effectively simulating the process of sound-signal transmission to inner ear hair cells and subsequent modulation filtering by neural cells. Secondly, building upon the observed masking effects in the human auditory system, we enhance our auditory-inspired modulation encoder by incorporating a masking mechanism resulting in the AMME. The AMME amplifies cleaner speech frequencies while suppressing noise components. Thirdly, inspired by the human auditory mechanism and capitalizing on contextual information, we leverage the attention mechanism for VAD. This methodology uses an attention mechanism to assign higher weights to contextual information containing richer and more informative cues. Through extensive experimentation and evaluation, we demonstrated the superior performance of AMME-CANet in enhancing VAD under challenging noise conditions.

IIOF: Intra- and Inter-feature orthogonal fusion of local and global features for music emotion recognition

In this paper, we propose intra- and inter-feature orthogonal fusion (IIOF) of local and global features obtained from MS-SincResNet or MS-SSincResNet (a variant of MS-SincResNet) for music emotion recognition (MER). Given a raw waveform of music signal, MS-SincResNet/MS-SSincResNet is first used to learn several 2D representations having different receptive fields and obtain embeddings with time-frequency information from different layers. Then, local and global features are extracted from these embeddings. IIOF consisting of intra-feature OF and inter-feature OF is further employed to integrate both local and global features to obtain a discriminative descriptor for MER. The intra-feature OF is used to enhance the diversity of the global feature, and the inter-feature OF is utilized to reduce redundancies and produce complementary information between local and global features. The experimental results have demonstrated that the representation discriminability can be enhanced by IIOF considering the feature orthogonality. Furthermore, extensive experimental results have shown that the proposed method outperforms other state-of-the-art methods in terms of regression and classification tasks on the well-known MER datasets, including the DEAM dataset and the PMEmo dataset. The codes are available at https://github.com/PeiChunChang/MS-SSincResNet_with_IIOF.

An Overview on Sound Features in Time and Frequency Domain

Abstract Sound is the result of mechanical vibrations that set air molecules in motion, causing variations in air pressure that propagate as pressure waves. Represented as waveforms, these visual snapshots of sound reveal some of its characteristics. While waveform analysis offers limited insights, audio features provide a quantitative and structured way to describe sound, enabling data-driven analysis and interpretation. Different audio features capture various aspects of sound, facilitating a comprehensive understanding of the audio data. By leveraging audio features, machine learning models can be trained to recognize patterns, classify sounds, or make predictions, enabling the development of intelligent audio systems. Time-domain features, e.g., amplitude envelope, capture events from raw audio waveforms. Frequency domain features, like band energy ratio and spectral centroid, focus on frequency components, providing distinct information. In this paper, we will describe three time-domain and three frequency-domain features that we consider crucial and widely used. We will illustrate the suitability of each feature for specific tasks and draw general conclusions regarding the significance of sound features in the context of machine learning.

Speech Enhancement using Convolutional Autoencoder Network

Abstract—We present an end-to-end deep learning approach to denoising speech signals by processing the raw waveform directly. Given input audio containing speech corrupted by an additive background signal, the system aims to produce a processed signal that contains only the speech content. Recent approaches have shown promising results using various deep network architec- tures. In this paper, we propose to train a fully-convolutional context aggregation network using a deep feature loss. That loss is based on comparing the internal feature activations in a different network, trained for acoustic environment detection and domestic audio tagging. Our approach outperforms the stateof- the-art in objective speech quality metrics and in large-scale perceptual experiments with human listeners. It also outperforms an identical network trained using traditional regression losses. The advantage of the new approach is particularly pronounced for the hardest data with the most intrusive background noise, for which denoising is most needed and most challenging. Index Terms—speech enhancement,Fully convolutional denois- ing autoencoders, single channel audio source separation, stacked convolutional autoencoders, deep convolutional neural networks, deep learning.

Nocturnal Autonomic and Vasoconstriction Parameters By Wearable Wrist Photoplethysmography Predict Intensity of Imminent Pain in Subjects with Sickle Cell Disease

Introduction: Sudden and unpredictable onset of pain is one of the biggest contributors to the morbidity of sickle cell disease (SCD), thought to occur due to obstruction of microvascular blood flow by rigid sickle-shaped red blood cells. Autonomic mediated vasoconstriction in the peripheral vasculature, with resulting decrease in tissue perfusion, is likely a triggering mechanism for the onset of vaso-occlusive pain, while the autonomic nervous system is also implicated in the pathogenesis of chronic neuropathic pain. Our analysis of 212 polysomnograms in SCD subjects showed that greater magnitude of nocturnal vasoconstriction (Mvasoc) is predictive of increased frequency of hospitalizations for vaso-occlusive pain crises (VOC) in the subsequent years (Chalacheva et al., Am J Hematol, 2020). In order to further develop this finding into a biomarker of more imminent pain, we monitored nocturnal photoplethysmography (PPG) at home with a wearable wristband device to measure vasoconstriction activity and heart rate variability, along with a daily electronic pain survey. Methods: 10 subjects with SCD were consented and underwent autonomic monitoring with a wristband device (Biostrap Inc) that uses an optical sensor to measure continuous PPG during sleep, with the raw PPG waveforms available from a cloud-based server. A unique biomarker (Mvasoc) was used to quantify the magnitude of vasoconstriction from the raw PPG signal, considering the frequency, magnitude, and duration of autonomic mediated spontaneous vasoconstrictions each night. Heart rate variability was derived from the pulse-to-pulse interval (PPI) of the PPG signal. Spectral indices of the PPI represent parasympathetic activity (high frequency power; HFP) and sympatho-vagal balance (Low to high Ratio; LHR = low frequency power(LFP)/HFP). Subjects concurrently documented daily pain incidence using an electronic text-based REDCap survey, with pain intensity measured on a scale of 1-10 if experiencing pain. Daily stress levels were also recorded on a scale of 1-10. Multivariate regression analysis was performed. Results: The subjects had a mean age of 25 years (range 14- 40 years) with an average hemoglobin of 8.8 g/dl . A total of 68 usable nocturnal PPG recordings were made (3-16 nights per subject). Therewere 25 incidents of pain on the day following autonomic monitoring with Biostrap, with the pain intensity ranging from 3-8. Multivariate analysis of nocturnal autonomic and vasoconstriction parameters revealed that a lower HFP ( p =0.0002) and a higher Mvasoc ( p=0.02) both preceded a higher intensity of pain the following day (Table 1). These results are consistent with a greater parasympathetic withdrawal and a higher magnitude of peripheral vasoconstriction predicting higher intensity of imminent pain. Age, gender, hemoglobin, stress levels and LHR were not predictive of pain the following day. Conclusion: Augmented nocturnal vasoconstriction and parasympathetic withdrawal noted prior to high intensity of pain imply that dysautonomia, with attendant changes in peripheral perfusion, has a significant role in pathogenesis of pain in SCD. Furthermore, autonomic vasoreactivity parameters were not significantly altered on the night after pain onset suggesting that the dysautonomia precedes pain. These results support the possibility of employing indices of nocturnal autonomic vasoreactivity as a predictive biomarker for imminent pain in SCD and provides a potential window for therapeutic intervention. This pilot study demonstrates the successful use of a wearable device PPG to remotely monitor nocturnal autonomic vasoreactivity and the ability to objectively quantify peripheral vasoconstriction responses. These findings will need to be applied in a larger cohort of SCD subjects with longer term monitoring for further validation and development of autonomic vasoreactivity as a predictive tool for pain in SCD.

Abstract 15854: Pulmonary Arterial Waveform Analysis and Supervised Machine Learning Predicts Outcomes in Heart Failure: An Analysis of the CHAMPION Trial

Introduction: Pulmonary artery (PA) pressure monitors have been shown to prevent hospitalizations in heart failure patients using mean pressure values. PA waveform analysis offers an overall assessment of right ventricular load that combines resistance and compliance. Hypothesis: We hypothesised that clinical outcomes can be predicted from PA waveform analyses from patients with an implanted PA monitor. Methods: The CHAMPION trial database was analysed. Information about the demographics and data collection methods for this cohort have been published previously. All waveforms were analysed automatically using Python. Tau or RC time, respiratory rate, respiratory range, P asymptote, mean pulmonary arterial pressure (mPAP), diastolic pulmonary arterial pressure (dPAP) and systolic pulmonary arterial pressure (sPAP) were derived from the raw waveforms. Tau and P asymptote were calculated using phase plane analysis. Outcomes of hospitalization or mortality were adjudicated as part of the original trial. Statistics were performed in R. A decision tree statistical approach using the XGBoost algorithm was used. Contribution to the model was assessed using Shapley Additive Explanations (SHAP) values. The dataset was split into 70% train and 30% test sets. Results: 296,948 PA pressure traces from 547 patients were analysed. 3 patients had insufficient quality of trace to analyse. There were 1340 outcome events throughout the follow up period, median 664 days. The trained algorithm was able to predict clinical outcomes in the test dataset with 100% accuracy. The most significant contributors according the mean SHAP value were RC time (SHAP = 1.1), respiratory range (SHAP = 1.1) and heart rate (SHAP = 1.1). Conclusions: Detailed analysis of the PA waveform from PA pressure monitors is feasible and yields clinically relevant information. A supervised machine learning algorithm trained on this data can accurately predict hospitalization or mortality events in heart failure patients.

Abstract 18640: Universal Interpretation of Electrocardiograms as Plain Language Using Deep Learning

Introduction: Deep learning models have shown promise in diagnosing diseases using electrocardiogram (ECG) data. However, each condition currently requires a separate model, which increases computational burden and operational complexity while decreasing overall clinical utility. Objective: Development of a unified modeling approach that allows for the interpretation of any ECG as plain language. Methods: We extracted ECG data from five facilities within the Mount Sinai Health System in New York City using the GE MUSE system. These data contained raw ECG waveforms, as well as cardiologist-confirmed diagnoses situated as plain language. Waveform data were plotted to images, and available diagnoses were allotted unique identifiers using Natural Language Processing. We also considered echocardiographically derived parameters such as left ventricular ejection fraction, valvular disease, and hypertrophic cardiomyopathy which were available for a subset of these patients as additional diagnostic endpoints to create a list of diagnoses per ECG.We created an end-to-end transformer based neural network consisting of a pre-trained vision transformer for extracting features from the ECG joined to a generative transformer for predicting associated diagnostic text. Model performance was evaluated by checking for concordance between predicted and actual diagnoses. Results: A total of 8.5 million ECGs were used for pre-training the feature extracting vision transformer. A total of 152 diagnoses were embedded with this corpus of data within the ECG file. We analyzed 406,000 echocardiogram reports to generate additional diagnostic labels for pathologies mentioned above. The model correctly predicted all diagnoses for 86.7% cases, and at least one correct diagnosis for 96.2% cases. Conclusions: A unified deep learning model can effectively interpret a broad range of disease conditions from ECG and echo data. Such an approach can speed up physician workflows, and allow for easier screening for conditions not normally diagnosed from the ECG. Future work will focus on extending the vocabulary of the model.

Abstract 15965: Electrocardiogram-Based Artificial Intelligence Predicts Incident Heart Failure

Introduction: Heart failure (HF) is a major cause of cardiovascular morbidity. Efficient and accurate prediction of incident heart failure episodes may enable preventive efforts. Methods: Using raw 12-lead ECG waveforms, we developed a convolutional neural network to predict HF admissions (Electrocardiogram to Heart Failure, “ECG2HF”). ECG2HF was developed in 91,573 individuals in Massachusetts General Hospital (MGH), and validated in three independent test sets: MGH, Brigham and Women’s Hospital (BWH), and the UK Biobank, among individuals without a prior HF episode. HF events were identified using a validated natural language processing model (MGH and BWH), and inpatient diagnosis codes (UK Biobank). Discrimination was quantified using time-dependent average precision (AP) and area under the receiver operating characteristic curve (AUROC). Among individuals with available data, we compared ECG2HF to the Pooled Cohort Equations for Heart Failure (PCE-HF). Results: The test sets comprised MGH (15,695 individuals, age 55±17, 51% women), BWH (52,124 individuals, age 55±17, 59% women), and UK Biobank (42,135 individuals, age 65±8, 52% women). ECG2HF consistently discriminated incident HF events (MGH 10-year AP: 0.17 [95% CI 0.15-0.20], 10-year AUROC 0.88 [0.86-0.89]; BWH: 0.15 [0.14-0.17], 0.87 [0.85-0.88]; UK Biobank 3-year AP 0.036 [0.024-0.057], 3-year AUROC 0.73 [0.68-0.77]). ECG2HF risk consistently stratified longitudinal HF incidence. Compared to PCE-HF, discrimination using ECG2HF was favorable in MGH/BWH (improvement in AP 0.07 [-0.11-0.14], improvement in AUROC 0.13 [0.088-0.18]), and UK Biobank (improvement in AP 0.016 [0.0060-0.033], improvement in AUROC -0.0093 [-0.062-0.044]). Conclusions: Artificial intelligence-enabled analysis of the 12-lead ECG may facilitate identification of individuals at risk for HF to enable early treatment to reduce HF-related morbidity.

Autism spectrum disorder detection using variable frequency complex demodulation of the electroretinogram

The early diagnosis of neurodevelopmental conditions such as autism spectrum disorder (ASD), is an unmet need. One difficulty is the identification of a biological signal that relates to the ASD phenotype. The electroretinogram (ERG) waveform has been identified as a possible signal that could categorize neurological conditions such as ASD. The ERG waveform is derived from the electrical activity of photoreceptors and retinal neurons in response to a brief flash of light and provides an indirect 'window' into the central nervous system. Traditionally, the waveform is analyzed in the time-domain, but more recently time-frequency spectrum (TFS) analysis of ERG has been successfully carried out using discrete wavelet transformation (DWT) to characterize the morphological features of the signal. In this study, we propose the use of a high resolution TFS technique, namely variable frequency complex demodulation (VFCDM), to decompose the ERG waveform based on two signal flash strengths to build machine learning (ML) models to categorize ASD. ERG waveforms from N = 217 subjects (71 ASD, 146 control), at two different flash strengths, 446 and 113 Troland seconds (Td.s), from both right and left eyes were included. We analyzed the raw ERG waveforms using DWT and VFCDM. We computed features from the TFSs and trained ML models such as Random Forest, Gradient Boosting, Support Vector Machine to classify ASD from controls. ML models were validated using a subject independent validation strategy, and we found that the ML models with VFCDM features outperformed models using the DWT, achieving an area under the receiver operating characteristics curve of 0.90 (accuracy = 0.81, sensitivity = 0.85, specificity = 0.78). We found that the higher frequency range (80–300 Hz) included more relevant information for classifying ASD compared to the lower frequencies. We also found that the stronger flash strength of 446 Td.s in the right eye provided the best classification result which supports VFCDM analysis of the ERG waveform as a potential tool to aid in the identification of the ASD phenotype.

PhysioKit: An Open-Source, Low-Cost Physiological Computing Toolkit for Single- and Multi-User Studies.

The proliferation of physiological sensors opens new opportunities to explore interactions, conduct experiments and evaluate the user experience with continuous monitoring of bodily functions. Commercial devices, however, can be costly or limit access to raw waveform data, while low-cost sensors are efforts-intensive to setup. To address these challenges, we introduce PhysioKit, an open-source, low-cost physiological computing toolkit. PhysioKit provides a one-stop pipeline consisting of (i) a sensing and data acquisition layer that can be configured in a modular manner per research needs, and (ii) a software application layer that enables data acquisition, real-time visualization and machine learning (ML)-enabled signal quality assessment. This also supports basic visual biofeedback configurations and synchronized acquisition for co-located or remote multi-user settings. In a validation study with 16 participants, PhysioKit shows strong agreement with research-grade sensors on measuring heart rate and heart rate variability metrics data. Furthermore, we report usability survey results from 10 small-project teams (44 individual members in total) who used PhysioKit for 4-6 weeks, providing insights into its use cases and research benefits. Lastly, we discuss the extensibility and potential impact of the toolkit on the research community.

IIRI-Net: An interpretable convolutional front-end inspired by IIR filters for speaker identification

Learning interpretable filters in Convolutional Neural Networks (CNNs) is an approach that helps to build models with better generalization ability. Interpretable filters can reveal some hidden aspects of the task and help to improve the model. One of the most successful approaches in the field of the speech processing is SincNet, where the model learns some band-pass filters in the first layer of a CNN with a raw waveform as its input. In this paper, similar to SincNet, some meaningful filters are proposed, which here are inspired by Infinite Impulse Response (IIR) filters. The proposed model uses a phase correction process to ensure that phase linearity is satisfied. The effective length of the truncated IIR filter is calculated based on the accumulated energy, and the effect of changing the filter size on the final results has been investigated. The proposed model is evaluated in the speaker identification task on the TIMIT and Librispeech datasets and compared with traditional CNNs and four interpretable kernel-based models. The experimental results show the superiority of the proposed model both in performance and convergence speed. Moreover, some patterns of the speech signal, which lead to uniquely identifying a speaker, are analyzed by examining the spectrum of the learned filters.

Data-Driven Models for Sub-Cycle Dynamic Response of Inverter-Based Resources Using WMU Measurements

Using real-world data from Waveform Measurement Units (WMUs), this paper proposes novel data-driven methods to model the dynamic response of inverter-based resource (IBR) to the high-frequency disturbances that occur in practice in power systems. WMUs are an emerging class of smart grid sensors. They can capture the fast sub-cycle dynamics in power systems, which are overlooked by phasor measurement units (PMUs). After extracting the differential voltage and current waveforms from the raw waveform data, we develop multiple methods that include data-driven model library construction and proper model selection. One class of methods is proposed in frequency domain, which is based on modal analysis. Another class of methods is proposed in time domain, which is based on regression analysis of time-series. Experimental results based on real-world WMU data demonstrate the of performance the proposed methods.

End-to-End Speaker Profiling Using 1D CNN Architectures and Filter Bank Initialization

The automatic estimation of speaker characteristics, such as height, age, and gender, has various applications in forensics, surveillance, customer service, and many human-robot interaction applications. These applications are often required to produce a response promptly. This work proposes a novel approach to speaker profiling by combining filter bank initializations, such as continuous wavelets and gammatone filter banks, with one-dimensional (1D) convolutional neural networks (CNN) and residual blocks. The proposed end-to-end model goes from the raw waveform to an estimated height, age, and gender of the speaker by learning speaker representation directly from the audio signal without relying on handcrafted and pre-computed acoustic features. The conducted experiments on the TIMIT dataset show that the proposed approach outperforms many previous studies on speaker profiling with a mean absolute error (MAE) of 5.18 and 4.91 cm in height estimation and MAE of 5.36 and 6.07 years in age estimation for males and females, respectively, and achieving an accuracy of 99.98% in gender prediction.

Investigation of self-supervised pre-trained models for classification of voice quality from speech and neck surface accelerometer signals

Prior studies in the automatic classification of voice quality have mainly studied the use of the acoustic speech signal as input. Recently, a few studies have been carried out by jointly using both speech and neck surface accelerometer (NSA) signals as inputs, and by extracting mel-frequency cepstral coefficients (MFCCs) and glottal source features. This study examines simultaneously-recorded speech and NSA signals in the classification of voice quality (breathy, modal, and pressed) using features derived from three self-supervised pre-trained models (wav2vec2-BASE, wav2vec2-LARGE, and HuBERT) and using a support vector machine (SVM) as well as convolutional neural networks (CNNs) as classifiers. Furthermore, the effectiveness of the pre-trained models is compared in feature extraction between glottal source waveforms and raw signal waveforms for both speech and NSA inputs. Using two signal processing methods (quasi-closed phase (QCP) glottal inverse filtering and zero frequency filtering (ZFF)), glottal source waveforms are estimated from both speech and NSA signals. The study has three main goals: (1) to study whether features derived from pre-trained models improve classification accuracy compared to conventional features (spectrogram, mel-spectrogram, MFCCs, i-vector, and x-vector), (2) to investigate which of the two modalities (speech vs. NSA) is more effective as input in the classification task with pre-trained model-based features, and (3) to evaluate whether the deep learning-based CNN classifier can enhance the classification accuracy in comparison to the SVM classifier. The results revealed that the use of the NSA input showed better classification performance compared to the speech signal. Between the features, the pre-trained model-based features showed better classification accuracies, both for speech and NSA inputs compared to the conventional features. The two classifiers performed equally well for all the pre-trained model-based features for both speech and NSA signals. It was also found that the HuBERT features performed better than the wav2vec2-BASE and wav2vec2-LARGE features for both speech and NSA inputs. In particular, when compared to the conventional features, the HuBERT features showed an absolute accuracy improvement of 3%–6% for speech and NSA signals in the classification of voice quality.

Degramnet: effective audio analysis based on a fully learnable time–frequency representation

Current state-of-the-art audio analysis algorithms based on deep learning rely on hand-crafted Spectrogram-like audio representations, that are more compact than descriptors obtained from the raw waveform; the latter are, in turn, far from achieving good generalization capabilities when few data are available for the training. However, Spectrogram-like representations have two main limitations: (1) The parameters of the filters are defined a priori, regardless of the specific audio analysis task; (2) such representations do not perform any denoising operation on the audio signal, neither in the time domain nor in the frequency domain. To overcome these limitations, we propose a new general-purpose convolutional architecture for audio analysis tasks that we call DEGramNet, which is trained with audio samples described with a novel, compact and learnable time–frequency representation that we call DEGram. The proposed representation is fully trainable: Indeed, it is able to learn the frequencies of interest for the specific audio analysis task; in addition, it performs denoising through a custom time–frequency attention module, which amplifies the frequency and time components in which the sound is actually located. It implies that the proposed representation can be easily adapted to the specific problem at hands, for instance giving more importance to the voice frequencies when the network needs to be used for speaker recognition. DEGramNet achieved state-of-the-art performance on the VGGSound dataset (for Sound Event Classification) and comparable accuracy with a complex and special-purpose approach based on network architecture search over the VoxCeleb dataset (for Speaker Identification). Moreover, we demonstrate that DEGram allows to achieve high accuracy with lightweight neural networks that can be used in real-time on embedded systems, making the solution suitable for Cognitive Robotics applications.