Speech Signal Research Articles

Speech Recognition for Air Traffic Control Utilizing a Multi-Head State-Space Model and Transfer Learning

Speech Recognition for Air Traffic Control Utilizing a Multi-Head State-Space Model and Transfer Learning

Pushing the Limits of Acoustic Spatial Perception via Incident Angle Encoding

With the growing popularity of smart speakers, numerous novel acoustic sensing applications have been proposed for low-frequency human speech and high-frequency inaudible sounds. Spatial information plays a crucial role in these acoustic applications, enabling various location-based services. However, typically commercial microphone arrays face limitations in spatial perception of inaudible sounds due to their sparse array geometries optimized for low-frequency speech. In this paper, we introduce MetaAng, a system designed to augment microphone arrays by enabling wideband spatial perception across both speech signals and inaudible sounds by leveraging the spatial encoding capabilities of acoustic metasurfaces. Our design is grounded in the fact that, while sensitive to high-frequency signals, acoustic metasurfaces are almost non-responsive to low-frequency speech due to significant wavelength discrepancy. This observation allows us to integrate acoustic metasurfaces with sparse array geometry, simultaneously enhancing the spatial perception of high-frequency and low-frequency acoustic signals. To achieve this, we first utilize acoustic metasurfaces and a configuration optimization algorithm to encode the unique features for each incident angle. Then, we propose an unrolling soft thresholding network that employs neural-enhanced priors and compressive sensing for high-accuracy, high-resolution multi-source angle estimation. We implement a prototype, and experimental results demonstrate that MetaAng maintains robustness across various scenarios, facilitating multiple applications, including localization and tracking.

New two-microphone simplified sub-band forward algorithm based on separated variable step-sizes for acoustic noise reduction

This paper presents an extended two-microphone adaptive filtering algorithm implemented on the sub-band simplified forward technique using new separated variable step-sizes parameters (denoted SSF-SVS: Simplified Sub-band Forward Separated Variable Step-sizes). This algorithm is proposed to reduce the acoustic noise and enhance the quality of estimated speech signal. Firstly, a sub-signal decomposition processing strategy is used to facilitating effective noise reduction with very fast convergence rate using two analysis filter banks. Secondly, a two-microphone simplified forward blind source separation (BSS) structure updated by the normalized least-mean-square algorithm (NLMS) for extracting the estimated speech signal. To guarantee a very fast convergence rate with high level of enhanced speech quality, thirdly we proposed new separated variable step-sizes parameters for controlling the adaptation of sub-band filters. The concept involves suggesting individual variable step-size parameter for each adaptive sub-filter, which are updated independently using the input sub-signal and the estimated output sub-signal. Experimental results confirm significant improvements in terms of convergence rate based on the mean square error criterion (MSE) in diverse acoustical environments. In other hand, the good performance of the proposed SSF-SVS algorithm in terms of speech quality is verified by on output signal-to-noise ratio (Output-SNR). The proposed algorithm compared with six two-microphone adaptive filtering algorithms in term of convergence rate, speech quality and computational complexity.

Flexible Self-Powered Low-Decibel Voice Recognition Mask

In environments where silent communication is essential, such as libraries and conference rooms, the need for a discreet means of interaction is paramount. Here, we present a single-electrode, contact-separated triboelectric nanogenerator (CS-TENG) characterized by robust high-frequency sensing capabilities and long-term stability. Integrating this TENG onto the inner surface of a mask allows for the capture of conversational speech signals through airflow vibrations, generating a comprehensive dataset. Employing advanced signal processing techniques, including short-time Fourier transform (STFT), Mel-frequency cepstral coefficients (MFCC), and deep learning neural networks, facilitates the accurate identification of speaker content and verification of their identity. The accuracy rates for each category of vocabulary and identity recognition exceed 92% and 90%, respectively. This system represents a pivotal advancement in facilitating secure and efficient unobtrusive communication in quiet settings, with promising implications for smart home applications, virtual assistant technology, and potential deployment in security and confidentiality-sensitive contexts.

A joint feature refining and multi-resolution feature learning approach for bandwidth extension of speech coded by EVS codec

The bandwidth of the speech signals is often limited in speech communications due to the specifications of the standardized codecs or insufficient bitrates. So, we present a bandwidth extension (BWE) framework of speech signals coded by the enhanced voice services (EVS) codec. Previous studies on speech bandwidth extension based on convolutional neural network (CNN) which exist channel and spatial redundancy feature information. This work proposes an end-to-end architecture which combines novel channel and spatial reconstruction module. Specifically, the spatial reconstruction module utilizes a mask to distinguish high-frequency and low-frequency features. Group convolutions is then employed to enhance high-frequency features and suppress low-frequency features. The channel reconstruction module is introduced to reduce unnecessary feature information. Additionally, we introduce a novel time–frequency loss function, incorporating time-domain loss and frequency-domain loss based on Mel-spectrum and multi-resolution representation, to optimize the network. To assess the model and loss function’s performance, we conducted experiments on subdatasets encoded at rates of 6.6kbps, 7.2kbps, and 8kbps, respectively. The experimental results demonstrated that our proposed model surpassed other baseline models in terms of LSD, SNR, PESQ, and MOS scores.

Exploration of Speech and Music Information for Movie Genre Classification

Movie genre prediction from trailers is mostly attempted in a multi-modal manner. However, the characteristics of movie trailer audio indicate that this modality alone might be highly effective in genre prediction. Movie trailer audio predominantly consists of speech and music signals in isolation or overlapping conditions. This work hypothesizes that the genre labels of movie trailers might relate to the composition of their audio component. In this regard, speech-music confidence sequences for the trailer audio are used as a feature. In addition, two other features previously proposed for discriminating speech-music are also adopted in the current task. This work proposes a time and channel Attention Convolutional Neural Network (ACNN) classifier for the genre classification task. The convolutional layers in ACNN learn the spatial relationships in the input features. The time and channel attention layers learn to focus on crucial time steps and CNN kernel outputs, respectively. The Moviescope dataset is used to perform the experiments, and two audio-based baseline methods are employed to benchmark this work. The proposed feature set with the ACNN classifier improves the genre classification performance over the baselines. Moreover, decent generalization performance is obtained for genre prediction of movies with different cultural influences (EmoGDB).

Amharic spoken digits recognition using convolutional neural network

Spoken digits recognition (SDR) is a type of supervised automatic speech recognition, which is required in various human–machine interaction applications. It is utilized in phone-based services like dialing systems, certain bank operations, airline reservation systems, and price extraction. However, the design of SDR is a challenging task that requires the development of labeled audio data, the proper choice of feature extraction method, and the development of the best performing model. Even if several works have been done for various languages, such as English, Arabic, Urdu, etc., there is no developed Amharic spoken digits dataset (AmSDD) to build Amharic spoken digits recognition (AmSDR) model for the Amharic language, which is the official working language of the government of Ethiopia. Therefore, in this study, we developed a new AmSDD that contains 12,000 utterances of 0 (Zaero) to 9 (zet’enyi) digits which were recorded from 120 volunteer speakers of different age groups, genders, and dialects who repeated each digit ten times. Mel frequency cepstral coefficients (MFCCs) and Mel-Spectrogram feature extraction methods were used to extract trainable features from the speech signal. We conducted different experiments on the development of the AmSDR model using the AmSDD and classical supervised learning algorithms such as Linear Discriminant Analysis (LDA), K-Nearest Neighbors (KNN), Support Vector Machine (SVM), and Random Forest (RF) as the baseline. To further improve the performance recognition of AmSDR, we propose a three layers Convolutional Neural Network (CNN) architecture with Batch normalization. The results of our experiments show that the proposed CNN model outperforms the baseline algorithms and scores an accuracy of 99% and 98% using MFCCs and Mel-Spectrogram features, respectively.

Features of the Recognition of Speech Signals in Conditions of Vocal Competition in Health and in Impairments to Auditory Speech Function

Features of the Recognition of Speech Signals in Conditions of Vocal Competition in Health and in Impairments to Auditory Speech Function

Codebook Optimization Using ACO Algorithm

The ACO algorithm, inspired by ant behavior, is used to optimize codebooks in speech processing. This study focuses on compressing speech signals efficiently to reduce bandwidth usage. Vector Quantization, a block coding technique, is employed, utilizing codebooks generated by the Linde-Buzo-Gray (LBG) algorithm. The ACO algorithm is proposed as an optimization method for these codebooks, aiming to minimize spectral distortion. Results show that ACO-optimized codebooks reduce spectral distortion. Keywords— Vector quantization, Speech compression, Linde-Buzo-Gray, ACO.

Cochlear-implant listeners benefit from training with time-compressed speech, even at advanced ages.

This study evaluated whether adaptive training with time-compressed speech produces an age-dependent improvement in speech recognition in 14 adult cochlear-implant users. The protocol consisted of a pretest, 5 h of training, and a posttest using time-compressed speech and an adaptive procedure. There were significant improvements in time-compressed speech recognition at the posttest session following training (>5% in the average time-compressed speech recognition threshold) but no effects of age. These results are promising for the use of adaptive training in aural rehabilitation strategies for cochlear-implant users across the adult lifespan and possibly using speech signals, such as time-compressed speech, to train temporal processing.

Intelligent speech elderly rehabilitation learning assistance system based on deep learning and sensor networks

Recently, deep learning has been proved to significantly improve the quality of speech recognition. Convolutional neural networks are often used in speech recognition tasks because of their special network structure and powerful learning function. In order to solve the problem that the traditional convolutional neural network can not reflect the one-dimensional basic attributes of speech signal, this paper proposes to set the number of frames for one dimension of convolution kernel, and use one-dimensional model and two-dimensional convolutional network model for speech recognition. By moving the convolution kernel of time axis and frequency band, it can adapt to the time change of speech signal and maintain the correlation between frequency bands to a great extent. At the same time, this paper also discusses the speech signal preprocessing, feature parameter extraction and regularization algorithm. Due to the lack of hospital resources, the lag of information technology, the poor ability of accompanying and other reasons, the current accompanying service for elderly rehabilitation can not meet the needs of elderly patients. The rapid development and wide use of information technology provide opportunities for the optimization of health services for the elderly. In view of the difficulty of word memory and interpretation in current rehabilitation courses, a VR intelligent teaching system based on intelligent voice technology is developed. The application results show that the system can effectively improve the ability of language expression and word writing. At present, the system of intelligent speech function has not been completed, and lacks speech synthesis function. The next research will focus on the use of speech synthesis technology, in order to realize the man-machine dialogue between people and the system, and show a more real training situation.

Gammatone-Frequency Cepstral Coefficients Based Fear Emotion Level Recognition System

Emotions represent affective states that induce alterations in behavior and interactions within one's environment. An avenue for discerning human emotions lies in the realm of speech analysis. Empirical evidence indicates that 1.6 million Indonesian teenagers grapple with mental anxiety disorders, characterized by sensations of fear or ambiguous vigilance. This work endeavors to devise a tool for discerning an individual's emotional state through voice processing, focusing particularly on fear emotions stratified into three levels of intensity: low, medium, and high. The proposed system employs Gammatone-Frequency Cepstral Coefficients (GFCC) for feature extraction, leveraging the efficacy of its gamma filter in reducing noise. Furthermore, a Random Forest (RF) Classifier is integrated to facilitate the recognition of fear's emotional intensity in speech signals. The system is deployed on a Raspberry Pi 4B and establishes a Bluetooth connection using the RFCOMM communication protocol to an Android application, presenting the classification results. The outcomes reveal that the Signal-to-Noise Reduction achieved through GFCC extraction surpasses that of Mel-Frequency Cepstral Coefficients (MFCC). In terms of accuracy, the implemented recognition system for fear emotion levels, employing GFCC extraction and Random Forest Classifier, attains a commendable accuracy of 73.33 %.

Squeeze-and-excitation 3D convolutional attention recurrent network for end-to-end speech emotion recognition

Speech emotion recognition (SER) is difficult since emotions are complex and dynamic processes involving multiple dimensions and sub-dimensions. Feature extraction is a challenging step in SER, where relevant features are extracted from the speech to identify emotional states accurately. Overcoming these challenges is essential to ensuring the effectiveness and robustness of the SER system. 3D-convolutional neural networks (3D-CNNs) are successfully used for feature extraction in SER. Speech signals can be converted into spectrogram-like representations where one axis represents time, another frequency, and the third can represent additional context or features. For modeling the dynamic nature of speech, a squeeze-and-excitation-based 3D-CNN model is employed for capturing temporal and spatial features of speech. Attention Gated Recurrent Units (AGRU) are applied to the extracted features for learning long-range temporal dependencies and selecting more informative features. The extraction and selection of the spatio-temporal feature representation lead to a hierarchical representation of the input speech. The fusion of squeeze-and-excitation 3D-CNN and AGRU (named SE3D-CARN) is evaluated on two datasets, EMO-BD and IEMOCAP, to identify various emotional states in speech. The proposed SER model reached an accuracy of 94.2% and 81.1% on the EMO-BD and IEMOCAP datasets, respectively.

Marathi Speech Emotion recognition using Deep Learning techniques.

Marathi Speech Emotion recognition using Deep Learning techniques.

IMAR in Frequency Domain for Separating Speech Signals by STFT

Blind Source Separation is a statistical method for isolating signals coming from unidentified sources that have been picked up by a variety of sensors. In the fields of biological signal analysis and speech analysis technology, it is an active researcher. The quantity of noise that is present in the signal throughout the speech signal separation process is a disadvantage. This noise affects the signal that has been separated, and it is often loud melodic noise. For the purpose of separating speech signals from BSS, the suggested method makes use of a network that is configured with the parameters of Instantaneous Mixing in addition to an Auto Regressive model and a maximum-likelihood function. In order to separate the individual voice signal from the input mixed signals, the Back Propagation Network is put into operation. The Short Time Fourier Transform is then utilized for the purpose of dividing the spoken stream into extremely precise time periods. Next, Maximum-likelihood method is implemented to determine the optimum values of the IMAR model's W and G parameters. This step is done after the ideal values have been estimated. The suggested model was tested using a combination of voice signals and microphone signals, and the results demonstrated that it provided superior execution in comparison to other algorithms that are currently in use.

Face and Speech Emotion Recognition System

Abstract: Emotions serve as the cornerstone of human communication, facilitating the expression of one's inner thoughts and feelings to others. Speech Emotion Recognition (SER) represents a pivotal endeavour aimed at deciphering the emotional nuances embedded within a speaker's voice signal. Universal emotions such as neutrality, anger, happiness, and sadness form the basis of this recognition process, allowing for the identification of fundamental emotional states. To achieve this, spectral and prosodic features are leveraged, each offering unique insights into the emotional content of speech. Spectral features, exemplified by the Mel Frequency Cepstral Coefficient (MFCC), provide a detailed analysis of the frequency distribution within speech signals, while prosodic features encompass elements like fundamental frequency, volume, pitch, speech intensity, and glottal parameters, capturing the rhythmic and tonal variations indicative of different emotional states. Through the integration of these features, SER systems can effectively simulate and classify a diverse range of emotional expressions, paving the way for enhanced human-computer interaction and communication technologies

Speech emotion recognition using a combination of variational mode decomposition and Hilbert transform

The objective of automated speech emotion recognition (SER) is to recognize each emotion of the speech signal uniquely and efficiently using machines like computers, mobile devices, etc. The popularity of SER has been widely acknowledged among researchers due to its extensive applicability in practical contexts. The use of SER has shown to be advantageous in several domains, including medical treatment, enhancement of security systems, surveillance operations, online marketing strategies, online educational platforms, online search engines, personal communication, customer relationship management, reinforcement of machine and human connection, and numerous other areas. Numerous authors have used various techniques, including the combination of multiple features (acoustic, non-acoustic, or both) and classifiers (machine learning, deep learning, or both), in order to enhance the efficacy of emotion categorization. In the present work, our objective is to enhance the performance of emotion classification (PECL) by using a combined approach of the variational mode decomposition (VMD) and Hilbert transform (HT) techniques. Using the VMD method, we decomposed the speech signal frame into many sub-signals, or intrinsic mode functions (IMFs). Then HT is applied to each VMD-based IMF signal to find the mode instantaneous amplitude (MIA) and mode instantaneous frequency (MIF) signal vectors. We extracted proposed features such as HT-based approximate entropy (HTAE), HT-based permutation entropy (HTPE), HT-based increment entropy (HTIE), and HT-based sample entropy (HTSE) using each MIA and MIF signal vector. The combination of the proposed HT-based features is called HT-based entropy (HTE) features. Then, we accessed the PECL using the HTE and MFCC features alone and in conjunction with a deep neural network (DNN) classifier. The experiment's results showed that the combinations of the proposed feature (MFCC + HTE) using a DNN classifier outperformed the individual features and obtained a SER accuracy of 86.92% for the EMOVO dataset and 91.63% for the EMO-DB dataset.

Emotion Analysis and Expression Algorithm of Dance Action Based on Machine Learning

Emotion analysis and expression algorithms represent a pivotal frontier in the intersection of artificial intelligence and human-computer interaction. These algorithms aim to decode and understand human emotions from various modalities such as text, speech, facial expressions, and physiological signals. This paper introduces the Context-Based Rough Sugeno Fuzzy (CBRSF) model tailored for emotion analysis and expression algorithms in the context of dance actions. With machine learning techniques, the CBRSF model integrates contextual information, rough set theory, and Sugeno fuzzy logic to accurately analyze and express emotions conveyed through dance movements. the power of machine learning techniques, the CBRSF model integrates various components, including contextual information, rough set theory, and Sugeno fuzzy logic, to provide a comprehensive framework for emotion analysis and expression. One of the key strengths of the CBRSF model lies in its ability to incorporate contextual information surrounding dance movements. Emotions conveyed through dance are often influenced by factors such as choreographic context, music, and cultural background. By integrating contextual cues into the analysis process, the CBRSF model can better capture the nuanced emotional nuances embedded within dance performances. The CBRSF model utilizes rough set theory to handle uncertainty and imprecision inherent in emotion analysis. Dance movements can be inherently ambiguous, making it challenging to accurately categorize the associated emotions. Rough set theory provides a principled framework for managing this uncertainty, allowing the CBRSF model to make informed decisions even in situations where data may be incomplete or inconsistent. Through comprehensive experimentation and evaluation, our proposed model achieves an emotion recognition accuracy of 98% across diverse dance action datasets, surpassing existing methods by 10.2%. Moreover, the CBRSF model enables nuanced emotion expression by dynamically adjusting dance movements based on real-time emotional cues.

Deep Learning Backward Recognition Model to Improve Pronunciation Accuracy in the Teaching of Spoken English

The advent of speech recognition algorithms has opened new avenues for enhancing pronunciation accuracy in the teaching of spoken English. As English continues to be a global lingua franca, proficiency in spoken communication holds paramount importance for individuals across diverse professional and academic domains. This paper introduces Forward Backward Recognition Deep Learning (FBRDL), a novel approach aimed at leveraging speech recognition algorithms to enhance pronunciation accuracy in the teaching of spoken English. FBRDL incorporates advanced deep learning techniques such as recurrent neural networks (RNNs) and transformers, which excel in modeling sequential data and capturing long-range dependencies. By leveraging these powerful architectures, FBRDL can effectively handle the inherent variability and complexity of speech signals, enabling robust and accurate recognition even in noisy or adverse environments. Moreover, FBRDL is characterized by its adaptability and scalability, making it well-suited for a wide range of applications across industries. Whether in the realm of virtual assistants, automatic transcription, or voice-controlled devices, FBRDL offers a versatile solution capable of meeting the demands of modern speech recognition tasks. FBRDL integrates principles from deep learning with advanced speech recognition techniques to provide learners with real-time feedback and guidance on their pronunciation. By analyzing spoken English inputs and identifying phonetic discrepancies, FBRDL offers targeted interventions tailored to individual learners' needs. FBRDL achieves an average increase in pronunciation accuracy of 20% compared to traditional teaching methods. Moreover, qualitative assessments underscore the effectiveness of FBRDL in facilitating more precise and efficient acquisition of spoken English skills.

Noise Robust VAD Technique and its Performance Evaluation

In this real world of audio and speech signals, the ability to detect the voice activity and remove the background voice is important. This project presents that to remove noise robust Voice Activity Detection (VAD) technique that combines voice source and vocal tract system information using a zero-frequency filtering (ZFF) approach. As mentioned earlier ZFF technique is employed to combine and to compute a composite signal that encapsulate essential parameters such as fundamental frequency and formats, enabling robust VAD in time domain.This methodology offers a significant advantage in terms of computational efficiency compared to other methods, making it an attractive choice for real-time applications. By applying dynamic thresholding after spectral entropy-based weighting, this approach exhibits resilience or minimum across a range of Signal-to-Noise Ratios (SNRs), further enhancing its ability.This project provides comprehensive utility for audio segmentation, which offers the capability to process individual audio files or entire directories. It also includes the extraction of a composite signal, which is a critical component in enhancing voice quality and reducing background noise. With this, the project is open to use build further and use it for the big industries purpose and for communication purpose. Keywords— Voice Activity Detection (VAD), Zero-Frequency Filtering (ZFF), Signal Noise Ratio (SNR).