Learning-based Speech Research Articles

Contrastive Learning based Speech Spoofing Detection for Multimedia Security in Edge Intelligence

Artificial intelligence (AI) empowered edge computing has given rise to a new paradigm and effectively facilitated the promotion and development of multimedia applications. The speech assistant is one of the significant services provided by multimedia applications, which aims to offer intelligent interactive experiences between humans and machines. However, malicious attackers may exploit spoofed speeches to deceive speech assistants, posing great challenges to the security of multimedia applications. The limited resources of multimedia terminal devices hinder their ability to effectively load speech spoofing detection models. Furthermore, processing and analyzing speech in the cloud can result in poor real-time performance and potential privacy risks. Existing speech spoofing detection methods rely heavily on annotated data and exhibit poor generalization capabilities for unseen spoofed speeches. To address these challenges, this paper first proposes the Coordinate Attention Network (CA2Net) that consists of coordinate attention blocks and Res2Net blocks. CA2Net can simultaneously extract temporal and spectral speech feature information and represent multi-scale speech features at a granularity level. Besides, a contrastive learning-based speech spoofing detection framework named GEMINI is proposed. GEMINI can be effectively deployed on edge nodes and autonomously learn speech features with strong generalization capabilities. GEMINI first performs data augmentation on speech signals and extracts conventional acoustic features to enhance the feature robustness. Subsequently, GEMINI utilizes the proposed CA2Net to further explore the discriminative speech features. Then, a tensor-based multi-attention comparison model is employed to maximize the consistency between speech contexts. GEMINI continuously updates CA2Net with contrastive learning, which enables CA2Net to effectively represent speech signals and accurately detect spoofed speeches. Extensive experiments on the ASVspoof2019 dataset show that GEMINI reduces the Equal Error Rate and tandem Detection Cost Function by up to 96.75% and 96.35% in the physical access scenario, and by up to 86.62% and 87.71% in the logical access scenario compared to peer methods.

Research on Deep Learning-based Speech Emotion Recognition System

Research on Deep Learning-based Speech Emotion Recognition System

Application of deep learning-based speech signal processing technology in electronic communication

In recent years, the artificial intelligence boom triggered by deep learning is influencing and changing peoples lifestyles. People are no longer satisfied with human-computer interaction through simple text commands; instead, they look forward to more convenient and faster communication methods like voice interaction. Against the backdrop of innovative development, the application of speech signal processing systems is becoming increasingly widespread. Therefore, it is necessary to study the application of deep learning-based speech signal processing technology in electronic communication. This can provide more valuable references and assistance for future development, promoting the better development of deep learning-based speech signal processing technology in electronic communication. In this paper, we first review the application of deep learning in speech signal enhancement, speech recognition, and speech synthesis from a theoretical analysis perspective. Then, we discuss the application of deep learning-based speech signal processing in electronic communication, including the application of models such as Transformer, LAS (Listen, Attend and Spell), and GFT-conformer in speech signal processing. We also discuss some application scenarios of deep learning-based speech signal processing in electronic communication. Finally, we identify the need for deeper application of deep learning technology in speech signal processing and electronic communication, with continuous optimization and adjustment.

Multi-language: ensemble learning-based speech emotion recognition

Multi-language: ensemble learning-based speech emotion recognition

A speech enhancement model based on noise component decomposition: Inspired by human cognitive behavior

In multimedia intelligent systems, speech enhancement is commonly employed to improve the quality of speech signals, making them clearer and more natural. Current deep learning-based speech enhancement models typically treat noise as a unified entity and aim to separate it from the target speech. In this paper, inspired by the cognitive behavior of the human brain when observing noisy speech spectrograms, we decompose the spectral energy of noise into regular and random components. We propose an auxiliary-model-based speech enhancement framework that better suppresses noise components closely resembling speech features. Firstly, we introduce a voiceprint segmentation network (VSnet) that partitions noisy speech into voiceprint and non-voiceprint regions. Subsequently, we present a noise reconstruction network (NRnet) that utilizes noise information from non-voiceprint regions to reconstruct and suppress the regular noise components within the voiceprint region. Finally, we construct a combination of a model dedicated to suppressing random components (RANnet) and a speech enhancement model (SEnet), and train them synchronously. By sharing encoder parameters, SEnet is compelled to reduce the extraction of regular noise features from the original noisy speech, contributing to improving speech quality generated through the decoder. Experimental results on public Voickbank-DEMAND and DNS-challenge 2020 datasets demonstrate that our approach achieves state-of-the-art performance.

Sub-convolutional U-Net with transformer attention network for end-to-end single-channel speech enhancement

Recent advancements in deep learning-based speech enhancement models have extensively used attention mechanisms to achieve state-of-the-art methods by demonstrating their effectiveness. This paper proposes a transformer attention network based sub-convolutional U-Net (TANSCUNet) for speech enhancement. Instead of adopting conventional RNNs and temporal convolutional networks for sequence modeling, we employ a novel transformer-based attention network between the sub-convolutional U-Net encoder and decoder for better feature learning. More specifically, it is composed of several adaptive time―frequency attention modules and an adaptive hierarchical attention module, aiming to capture long-term time-frequency dependencies and further aggregate hierarchical contextual information. Additionally, a sub-convolutional encoder-decoder model used different kernel sizes to extract multi-scale local and contextual features from the noisy speech. The experimental results show that the proposed model outperforms several state-of-the-art methods.

On phase recovery and preserving early reflections for deep-learning speech dereverberation.

In indoor environments, reverberation often distorts clean speech. Although deep learning-based speech dereverberation approaches have shown much better performance than traditional ones, the inferior speech quality of the dereverberated speech caused by magnitude distortion and limited phase recovery is still a serious problem for practical applications. This paper improves the performance of deep learning-based speech dereverberation from the perspectives of both network design and mapping target optimization. Specifically, on the one hand, a bifurcated-and-fusion network and its guidance loss functions were designed to help reduce the magnitude distortion while enhancing the phase recovery. On the other hand, the time boundary between the early and late reflections in the mapped speech was investigated, so as to make a balance between the reverberation tailing effect and the difficulty of magnitude/phase recovery. Mathematical derivations were provided to show the rationality of the specially designed loss functions. Geometric illustrations were given to explain the importance of preserving early reflections in reducing the difficulty of phase recovery. Ablation study results confirmed the validity of the proposed network topology and the importance of preserving 20 ms early reflections in the mapped speech. Objective and subjective test results showed that the proposed system outperformed other baselines in the speech dereverberation task.

A Two-Stage Deep Representation Learning-Based Speech Enhancement Method Using Variational Autoencoder and Adversarial Training

A Two-Stage Deep Representation Learning-Based Speech Enhancement Method Using Variational Autoencoder and Adversarial Training

Lightweight target speaker separation network based on joint training

Target speaker separation aims to separate the speech components of the target speaker from mixed speech and remove extraneous components such as noise. In recent years, deep learning-based speech separation methods have made significant breakthroughs and have gradually become mainstream. However, these existing methods generally face problems with system latency and performance upper limits due to the large model size. To solve these problems, this paper proposes improvements in the network structure and training methods to enhance the model’s performance. A lightweight target speaker separation network based on long-short-term memory (LSTM) is proposed, which can reduce the model size and computational delay while maintaining the separation performance. Based on this, a target speaker separation method based on joint training is proposed to achieve the overall training and optimization of the target speaker separation system. Joint loss functions based on speaker registration and speaker separation are proposed for joint training of the network to further improve the system’s performance. The experimental results show that the lightweight target speaker separation network proposed in this paper has better performance while being lightweight, and joint training of the target speaker separation network with our proposed loss function can further improve the separation performance of the original model.

A semi-supervised high-quality pseudo labels algorithm based on multi-constraint optimization for speech deception detection

Deep learning-based speech deception detection research relies on a large amount of labeled data. However, in the process of collecting speech deception detection data, the identification of truth and lies requires researchers to have a professional knowledge reserve, which greatly limits the number of annotated samples. Improving the accuracy of lie detection with insufficient annotation data is the focus of this study at this stage. In this paper, we propose a semi-supervised high-quality pseudo-label algorithm based on multi-constraint optimization (HQPL-MC) for speech deception detection. Firstly, the algorithm exploits the potential feature information of unlabeled data by using deep auto-encoder networks; secondly, it achieves entropy minimization with the help of the pseudo labeling technique to reduce the class overlap distribution of truth and deception data; finally, it improves the quality of pseudo labels by optimizing the unlabeled loss and reconstruction loss to further enhance the classification performance of the model when the labeled data is insufficient. We recorded an interview-style corpus by ourselves and used it in this paper for the experimental demonstration of the algorithm together with the Columbia/SRI/Colorado(CSC) corpus. The detection performance of the proposed algorithm is better than most state-of-the-art algorithms.

Speech Perception Improvement Algorithm Based on a Dual-Path Long Short-Term Memory Network.

Current deep learning-based speech enhancement methods focus on enhancing the time-frequency representation of the signal. However, conventional methods can lead to speech damage due to resolution mismatch problems that emphasize only specific information in the time or frequency domain. To address these challenges, this paper introduces a speech enhancement model designed with a dual-path structure that identifies key speech characteristics in both the time and time-frequency domains. Specifically, the time path aims to model semantic features hidden in the waveform, while the time-frequency path attempts to compensate for the spectral details via a spectral extension block. These two paths enhance temporal and spectral features via mask functions modeled as LSTM, respectively, offering a comprehensive approach to speech enhancement. Experimental results show that the proposed dual-path LSTM network consistently outperforms conventional single-domain speech enhancement methods in terms of speech quality and intelligibility.

Machine learning-based speech recognition system for nursing documentation – A pilot study

PurposeConsidering the significant workload of nursing tasks, enhancing the efficiency of nursing documentation is imperative. This study aimed to evaluate the effectiveness of a machine learning-based speech recognition (SR) system in reducing the clinical workload associated with typing nursing records, implemented in a psychiatry ward. MethodsThe study was conducted between July 15, 2020, and June 30, 2021, at Cheng Hsin General Hospital in Taiwan. The language corpus was based on the existing records from the hospital nursing information system. The participating ward's nursing activities, clinical conversation, and accent data were also collected for deep learning-based SR-engine training. A total of 21 nurses participated in the evaluation of the SR system. Documentation time and recognition error rate were evaluated in parallel between SR-generated records and keyboard entry over 4 sessions. Any differences between SR and keyboard transcriptions were regarded as SR errors.Findings: A total of 200 data were obtained from four evaluation sessions, 10 participants were asked to use SR and keyboard entry in parallel at each session and 5 entries were collected from each participant. Overall, the SR system processed 30,112 words in 32,456 s (0.928 words per second). The mean accuracy of the SR system improved after each session, from 87.06% in 1st session to 95.07% in 4th session. ConclusionThis pilot study demonstrated our machine learning-based SR system has an acceptable recognition accuracy and may reduce the burden of documentation for nurses. However, the potential error with the SR transcription should continually be recognized and improved. Further studies are needed to improve the integration of SR in digital documentation of nursing records, in terms of both productivity and accuracy across different clinical specialties.

A deep interpretable representation learning method for speech emotion recognition

This paper focuses on the active interpretability for deep learning-based speech emotion recognition (SER). To achieve this, we propose an explicit feature constrained model, the interpretable group convolutional neural network (IG-CNN) model. In the proposed model, we first introduce the interpretability constraint to learn human-understandable interpretable representations. The emotion prediction decision can be active interpreted via the model coefficients. To acquire more representations beyond interpretable ones, and ensure they are useful for SER, we then design the uncorrelation constraint between interpretable and autonomous representations and introduce group CNN structure. We test the model on IEMOCAP, RAVDESS, eNTERFACE’05, and CREMA-D datasets. Experimental results show that our model outperforms all the baselines. In addition, the proposed model can also learn the patterns of human perception of speech emotion and provide explanation for the recognition results.

Research on Speech Synthesis Based on Mixture Alignment Mechanism.

In recent years, deep learning-based speech synthesis has attracted a lot of attention from the machine learning and speech communities. In this paper, we propose Mixture-TTS, a non-autoregressive speech synthesis model based on mixture alignment mechanism. Mixture-TTS aims to optimize the alignment information between text sequences and mel-spectrogram. Mixture-TTS uses a linguistic encoder based on soft phoneme-level alignment and hard word-level alignment approaches, which explicitly extract word-level semantic information, and introduce pitch and energy predictors to optimally predict the rhythmic information of the audio. Specifically, Mixture-TTS introduces a post-net based on a five-layer 1D convolution network to optimize the reconfiguration capability of the mel-spectrogram. We connect the output of the decoder to the post-net through the residual network. The mel-spectrogram is converted into the final audio by the HiFi-GAN vocoder. We evaluate the performance of the Mixture-TTS on the AISHELL3 and LJSpeech datasets. Experimental results show that Mixture-TTS is somewhat better in alignment information between the text sequences and mel-spectrogram, and is able to achieve high-quality audio. The ablation studies demonstrate that the structure of Mixture-TTS is effective.

Multi-stage Progressive Learning-Based Speech Enhancement Using Time–Frequency Attentive Squeezed Temporal Convolutional Networks

Multi-stage Progressive Learning-Based Speech Enhancement Using Time–Frequency Attentive Squeezed Temporal Convolutional Networks

Application of deep learning-based speech system in online music learning system

Application of deep learning-based speech system in online music learning system

Speech enhancement approach for body-conducted unvoiced speech based on Taylor–Boltzmann machines trained DNN

Communication becomes effectual while the verbal communication signal reaches the destination with profound characteristics. An application based on human voice attracts the researcher's attention. It leads the researcher to proceed towards the speech conversion for low voice humans having murmured voices as their attitude. The researchers introduced a technique for improving the body-conducted unvoiced speech for silent speech communication using Taylor-Boltzmann machine-trained optimized Deep Neural Network (DNN). The projected model includes two main stages: the training stage as well as recognition stage. In the training stage, the following functionalities are undergone: (i) pre-processing, (ii) feature extraction, and (iii) deep learning-based speech enhancement. Initially, the collected input Non-Audible Murmur (NAM) speech signal is subjected to the pre-processing stage, wherein the improved wiener filtering is employed for suppressing the noise in the input signal. From the pre-processed speech signal, the most relevant characteristics such as “spectral skewness, spectral chroma, spectral centroid and improved Mel Frequency Cepstral Coefficients (MFCC)” are extracted, and they are fused together. Then, using these fused features, the Taylor series with optimized DNN and Boltzmann Machine (Taylor-DNN-BM) model is trained. The final enhanced speech is acquired from the Taylor- optimized DNN-BM. The recognition phase encapsulates the feature extraction and deep learning-based speech enhancement stage. The input speech data enters into the feature extraction phase, wherein the characteristics such as “spectral skewness, spectral chroma, spectral centroid and improved (MFCC)” are retrieved. After that, these features are fused and fed as input to the Taylor-optimized DNN-BM model. From Taylor- optimized DNN-BM, the enhanced speech signal is acquired. In DNN, the weight function is fine-tuned using the newly projected Self Improved Flower Pollination Algorithm (SI-FPA) algorithm. This SI-FPA model is the conceptual improvement of the Flower Pollination Algorithm (FPA). Finally, the projected model is compared with the existing models to validate its efficiency in terms of speech enhancement.

Deep Learning-Based Speech Analytics System for Stroke Risk Analysis in the Elderly

Deep Learning-Based Speech Analytics System for Stroke Risk Analysis in the Elderly

Deep convolutional neural networks-based features for Indonesian large vocabulary speech recognition

There are great interests in developing speech recognition using deep learning technologies due to their capability to model the complexity of pronunciations, syntax, and language rules of speech data better than the traditional hidden Markov model (HMM) do. But, the availability of large amount of data is necessary for deep learning-based speech recognition to be effective. While this is not a problem for mainstream languages such as English or Chinese, this is not the case for non-mainstream languages such as Indonesian. To overcome this limitation, we present deep features based on convolutional neural networks (CNN) for Indonesian large vocabulary continuous speech recognition in this paper. The CNN is trained discriminatively which is different from usual deep learning implementations where the networks are trained generatively. Our evaluations show that the proposed method on Indonesian speech data achieves 7.26% and 9.01% error reduction rates over the state-of-the-art deep belief networks-deep neural networks (DBN-DNN) for large vocabulary continuous speech recognition (LVCSR), with Mel frequency cepstral coefficients (MFCC) and filterbank (FBANK) used as features, respectively. An error reduction rate of 6.13% is achieved compared to CNN-DNN with generative training.

Deep learning-based speech recognition for Korean elderly speech data including dementia patients

Deep learning-based speech recognition for Korean elderly speech data including dementia patients