Real-time Speech Research Articles

MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach

Speech is the most dominant source of communication among humans, and it is an efficient way for human–computer interaction (HCI) to exchange information. Nowadays, speech emotion recognition (SER) is an active research area that plays a crucial role in real-time applications. In this era, the SER system has lacked real-time speech processing. To address this problem, we propose an end-to-end real-time SER model that is based on a one-dimensional dilated convolutional neural network (DCNN). Our model used a multi-learning strategy to parallel extract spatial salient emotional features and learn long term contextual dependencies from the speech signals. We used residual blocks with a skip connection (RBSC) module, in order to find a correlation, the emotional cues, and the sequence learning (Seq_L) module, to learn the long term contextual dependencies in the input features. Furthermore, we used a fusion layer to concatenate these learned features for the final emotion recognition task. Our model structure is quite simple, and it is capable of automatically learning salient discriminative features from the speech signals. We evaluated our model using benchmark IEMOCAP and EMO-DB datasets and obtained a high recognition accuracy, which were 73% and 90%, respectively. The experimental results indicated the significance and the efficiency of our proposed model have shown excessive assistance with the implementation of a real-time SER system. Hence, our model is capable of processing original speech signals for the emotion recognition that utilizes lightweight dilated CNN architecture that implements the multi-learning trick (MLT) approach.

Real Time Speech Recognition based on PWP Thresholding and MFCC using SVM

The real-time performance of Automatic Speech Recognition (ASR) is a big challenge and needs high computing capability and exhaustive memory consumption. Getting a robust performance against inevitable various difficult situations such as speaker variations, accents, and noise is a tedious task. It’s crucial to expand new and efficient approaches for speech signal extraction features and pre-processing. In order to fix the high dependency issue related to processing succeeding steps in ARS and enhance the extracted features’ quality, noise robustness can be solved within the ARS extraction block feature, removing implicitly the need for further additional specific compensation parameters or data collection. This paper proposes a new robust acoustic extraction approach development based on a hybrid technique consisting of Perceptual Wavelet Packet (PWP) and Mel Frequency Cepstral Coefficients (MFCCs). The proposed system was implemented on a Rasberry Pi board and its performance was checked in a clean environment, reaching 99% average accuracy. The recognition rate was improved (from 80% to 99%) for the majority of Signal-to-Noise Ratios (SNRs) under real noisy conditions for positive SNRs and considerably improved results especially for negative SNRs.

Real-time single-channel deep neural network-based speech enhancement on edge devices.

In this paper, we present a deep neural network architecture comprising of both convolutional neural network (CNN) and recurrent neural network (RNN) layers for real-time single-channel speech enhancement (SE). The proposed neural network model focuses on enhancing the noisy speech magnitude spectrum on a frame-by-frame process. The developed model is implemented on the smartphone (edge device), to demonstrate the real-time usability of the proposed method. Perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) test results are used to compare the proposed algorithm to previously published conventional and deep learning-based SE methods. Subjective ratings show the performance improvement of the proposed model over the other baseline SE methods.

Real-time speech enhancement algorithm for transient noise suppression

To effectively restrain stationary noise and transient noise, a real-time single-channel speech enhancement algorithm is proposed. First, to evaluate stationary noise, the quantile noise estimation method is used to obtain the spectrum of stationary noise. Then, based on the normalized variance and gravity center of the signal, the transient noise detection method is proposed to modify the spectrum of stationary noise. Next, the speech presence probability is estimated based on the speech features and harmonic analysis. Finally, the optimized-modified log-spectral amplitude (OM-LSA) estimator is adopted for speech enhancement. The experimental noise contains 115 environmental sounds with the SNR of −10 to 10 dB. The experimental results show that the performance of the proposed algorithm is comparable to the OM-LSA algorithm which has good denoising performance, but the real-time performance of the former is much better. Compared with the Webrtc real-time algorithm, under the overall performance of stationary noise and transient noise, the overall speech quality indicators of the improved algorithm increased by 7.5%, 7.8% and 5.0%, respectively. And the short-time objective intelligibility increased by 2.4%, 2.4% and 2.0%, respectively. Even compared with the recurrent neural network(RNN) algorithm, the suppression performance of the transient noise is better. Besides, the real-time experiment base on the hardware platform shows that the runtime of processing a 10 ms frame is 4.3 ms.

An Intelligent Mobile-Enabled System for Diagnosing Parkinson Disease: Development and Validation of a Speech Impairment Detection System.

BackgroundParkinson disease (PD) is one of the most common neurological diseases. At present, because the exact cause is still unclear, accurate diagnosis and progression monitoring remain challenging. In recent years, exploring the relationship between PD and speech impairment has attracted widespread attention in the academic world. Most of the studies successfully validated the effectiveness of some vocal features. Moreover, the noninvasive nature of speech signal–based testing has pioneered a new way for telediagnosis and telemonitoring. In particular, there is an increasing demand for artificial intelligence–powered tools in the digital health era.ObjectiveThis study aimed to build a real-time speech signal analysis tool for PD diagnosis and severity assessment. Further, the underlying system should be flexible enough to integrate any machine learning or deep learning algorithm.MethodsAt its core, the system we built consists of two parts: (1) speech signal processing: both traditional and novel speech signal processing technologies have been employed for feature engineering, which can automatically extract a few linear and nonlinear dysphonia features, and (2) application of machine learning algorithms: some classical regression and classification algorithms from the machine learning field have been tested; we then chose the most efficient algorithms and relevant features.ResultsExperimental results showed that our system had an outstanding ability to both diagnose and assess severity of PD. By using both linear and nonlinear dysphonia features, the accuracy reached 88.74% and recall reached 97.03% in the diagnosis task. Meanwhile, mean absolute error was 3.7699 in the assessment task. The system has already been deployed within a mobile app called No Pa.ConclusionsThis study performed diagnosis and severity assessment of PD from the perspective of speech order detection. The efficiency and effectiveness of the algorithms indirectly validated the practicality of the system. In particular, the system reflects the necessity of a publicly accessible PD diagnosis and assessment system that can perform telediagnosis and telemonitoring of PD. This system can also optimize doctors’ decision-making processes regarding treatments.

A Real-Time Dual-Microphone Speech Enhancement Algorithm Assisted by Bone Conduction Sensor.

The quality and intelligibility of the speech are usually impaired by the interference of background noise when using internet voice calls. To solve this problem in the context of wearable smart devices, this paper introduces a dual-microphone, bone-conduction (BC) sensor assisted beamformer and a simple recurrent unit (SRU)-based neural network postfilter for real-time speech enhancement. Assisted by the BC sensor, which is insensitive to the environmental noise compared to the regular air-conduction (AC) microphone, the accurate voice activity detection (VAD) can be obtained from the BC signal and incorporated into the adaptive noise canceller (ANC) and adaptive block matrix (ABM). The SRU-based postfilter consists of a recurrent neural network with a small number of parameters, which improves the computational efficiency. The sub-band signal processing is designed to compress the input features of the neural network, and the scale-invariant signal-to-distortion ratio (SI-SDR) is developed as the loss function to minimize the distortion of the desired speech signal. Experimental results demonstrate that the proposed real-time speech enhancement system provides significant speech sound quality and intelligibility improvements for all noise types and levels when compared with the AC-only beamformer with a postfiltering algorithm.

Real-time dual-channel speech enhancement by VAD assisted MVDR beamformer for hearing aid applications using smartphone.

In this paper, a dual-channel speech enhancement (SE) method is proposed. The proposed method is a combination of minimum variance distortionless response (MVDR) beamformer and a super-Gaussian joint maximum a posteriori (SGJMAP) based SE gain function. The proposed SE method runs on a smartphone in real-time, providing a portable device for hearing aid (HA) applications. Spectral Flux based voice activity detector (VAD) is used to improve the accuracy of the beamformer output. The efficiency of the proposed SE method is evaluated using speech quality and intelligibility measures and compared with that of other SE techniques. The objective and subjective test results show the capability of the proposed SE method in three different noisy conditions at low signal to noise ratios (SNRs) of -5, 0, and +5 dB.

An 8.93 TOPS/W LSTM Recurrent Neural Network Accelerator Featuring Hierarchical Coarse-Grain Sparsity for On-Device Speech Recognition

Long short-term memory (LSTM) is a type of recurrent neural networks (RNNs), which is widely used for time-series data and speech applications, due to its high accuracy on such tasks. However, LSTMs pose difficulties for efficient hardware implementation because they require a large amount of weight storage and exhibit computation complexity. Prior works have proposed compression techniques to alleviate the storage/computation requirements of LSTMs but elementwise sparsity schemes incur sizable index memory overhead and structured compression techniques report limited compression ratios. In this article, we present an energy-efficient LSTM RNN accelerator, featuring an algorithm-hardware co-optimized memory compression technique called hierarchical coarse-grain sparsity (HCGS). Aided by the HCGS-based blockwise recursive weight compression, we demonstrate LSTM networks with up to 16 $\times $ fewer weights while achieving minimal error rate degradation. The prototype chip fabricated in 65-nm LP CMOS achieves up to 8.93 TOPS/W for real-time speech recognition using compressed LSTMs based on HCGS. HCGS-based LSTMs have demonstrated energy-efficient speech recognition with low error rates for TIMIT, TED-LIUM, and LibriSpeech data sets.

A Lightweight Speaker Recognition System Using Timbre Properties

Speaker recognition is an active research area that contains notable usage in biometric security and authentication system. Currently, there exist many well-performing models in the speaker recognition domain. However, most of the advanced models implement deep learning that requires GPU support for real-time speech recognition, and it is not suitable for low-end devices. In this paper, we propose a lightweight text-independent speaker recognition model based on random forest classifier. It also introduces new features that are used for both speaker verification and identification tasks. The proposed model uses human speech based timbral properties as features that are classified using random forest. Timbre refers to the very basic properties of sound that allow listeners to discriminate among them. The prototype uses seven most actively searched timbre properties, boominess, brightness, depth, hardness, roughness, sharpness, and warmth as features of our speaker recognition model. The experiment is carried out on speaker verification and speaker identification tasks and shows the achievements and drawbacks of the proposed model. In the speaker identification phase, it achieves a maximum accuracy of 78%. On the contrary, in the speaker verification phase, the model maintains an accuracy of 80% having an equal error rate (ERR) of 0.24.

Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays

Signed languages are not as pervasive a conversational medium as spoken languages due to the history of institutional suppression of the former and the linguistic hegemony of the latter. This has led to a communication barrier between signers and non-signers that could be mitigated by technology-mediated approaches. Here, we show that a wearable sign-to-speech translation system, assisted by machine learning, can accurately translate the hand gestures of American Sign Language into speech. The wearable sign-to-speech translation system is composed of yarn-based stretchable sensor arrays and a wireless printed circuit board, and offers a high sensitivity and fast response time, allowing real-time translation of signs into spoken words to be performed. By analysing 660 acquired sign language hand gesture recognition patterns, we demonstrate a recognition rate of up to 98.63% and a recognition time of less than 1 s. Wearable yarn-based stretchable sensor arrays, combined with machine learning, can be used to translate American Sign Language into speech in real time.

A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach.

In the context of assisted human, identifying and enhancing non-stationary speech targets speech in various noise environments, such as a cocktail party, is an important issue for real-time speech separation. Previous studies mostly used microphone signal processing to perform target speech separation and analysis, such as feature recognition through a large amount of training data and supervised machine learning. The method was suitable for stationary noise suppression, but relatively limited for non-stationary noise and difficult to meet the real-time processing requirement. In this study, we propose a real-time speech separation method based on an approach that combines an optical camera and a microphone array. The method was divided into two stages. Stage 1 used computer vision technology with the camera to detect and identify interest targets and evaluate source angles and distance. Stage 2 used beamforming technology with microphone array to enhance and separate the target speech sound. The asynchronous update function was utilized to integrate the beamforming control and speech processing to reduce the effect of the processing delay. The experimental results show that the noise reduction in various stationary and non-stationary noise environments were 6.1 dB and 5.2 dB respectively. The response time of speech processing was less than 10ms, which meets the requirements of a real-time system. The proposed method has high potential to be applied in auxiliary listening systems or machine language processing like intelligent personal assistant.

Identifying Predictors of Levator Veli Palatini Muscle Contraction During Speech Using Dynamic Magnetic Resonance Imaging.

Purpose The purpose of this study was to identify predictors of levator veli palatini (LVP) muscle shortening and maximum contraction velocity in adults with normal anatomy. Method Twenty-two Caucasian English-speaking adults with normal speech and resonance were recruited. Participants included 11 men and 11 women (M = 22.8 years, SD = 4.1) with normal anatomy. Static magnetic resonance images were obtained using a three-dimensional static imaging protocol. Midsagittal and oblique coronal planes were established for visualization of the velum and LVP muscle at rest. Dynamic magnetic resonance images were obtained in the oblique coronal plane during production of "ansa." Amira 6.0.1 Visualization and Volume Modeling Software and MATLAB were used to analyze images and calculate LVP shortening and maximum contraction velocity. Results Significant predictors (p < .05) of maximum LVP shortening during velopharyngeal closure included mean extravelar length, LVP origin-to-origin distance, velar thickness, pharyngeal depth, and velopharyngeal ratio. Significant predictors (p < .05) of maximum contraction velocity during velopharyngeal closure included mean extravelar length, intravelar length, LVP origin-to-origin distance, and velar thickness. Conclusions This study identified six velopharyngeal variables that predict LVP muscle function during real-time speech. These predictors should be considered among children and individuals with repaired cleft palate in future studies.

Visual Speech Recognition: Improving Speech Perception in Noise through Artificial Intelligence.

To compare speech perception (SP) in noise for normal-hearing (NH) individuals and individuals with hearing loss (IWHL) and to demonstrate improvements in SP with use of a visual speech recognition program (VSRP). Single-institution prospective study. Tertiary referral center. Eleven NH and 9 IWHL participants in a sound-isolated booth facing a speaker through a window. In non-VSRP conditions, SP was evaluated on 40 Bamford-Kowal-Bench speech-in-noise test (BKB-SIN) sentences presented by the speaker at 50 A-weighted decibels (dBA) with multiperson babble noise presented from 50 to 75 dBA. SP was defined as the percentage of words correctly identified. In VSRP conditions, an infrared camera was used to track 35 points around the speaker's lips during speech in real time. Lip movement data were translated into speech-text via an in-house developed neural network-based VSRP. SP was evaluated similarly in the non-VSRP condition on 42 BKB-SIN sentences, with the addition of the VSRP output presented on a screen to the listener. In high-noise conditions (70-75 dBA) without VSRP, NH listeners achieved significantly higher speech perception than IWHL listeners (38.7% vs 25.0%, P = .02). NH listeners were significantly more accurate with VSRP than without VSRP (75.5% vs 38.7%, P < .0001), as were IWHL listeners (70.4% vs 25.0% P < .0001). With VSRP, no significant difference in SP was observed between NH and IWHL listeners (75.5% vs 70.4%, P = .15). The VSRP significantly increased speech perception in high-noise conditions for NH and IWHL participants and eliminated the difference in SP accuracy between NH and IWHL listeners.

Real-Time Speech Emotion Recognition Using a Pre-trained Image Classification Network: Effects of Bandwidth Reduction and Companding

This paper provides a step by step introduction to real-time speech emotion recognition (SER) using a pre-trained image classification network. The procedure has low computational requirements and can be implemented on different voice-based communication platforms such as mobile phones, call centers, and online communication facilities. Effects of reduced speech bandwidth and the mu-low companding procedure used in transmission systems on the SER accuracy are examined. The results showed that the baseline approach achieved an average accuracy of 82% when trained on the Berlin Emotional Speech (EMO-DB) data with seven categorical emotions. Reduction of the sampling frequency from the baseline 16 kHz to 8 kHz (i.e., bandwidth reduction from 8 kHz to 4 kHz respectively) led to a decrease of SER accuracy by about 3.3%. The companding procedure reduced the result by 3.8%, and the combined effect of both factors further reduced the average accuracy by about 7% compared to the baseline results. The SER was implemented in real-time with emotional labels generated every 1.033 to 1.026 seconds. Real-time implementation timelines are presented.

Real-Time S-T Analysis Method Based on Speaker Recognition

With the continuous deepening of curriculum reform, teaching evaluation has become an important part of investigating the effectiveness of classroom teaching. This study improved the original S-T analysis method with speaker recognition technology that tracks the speaker's identity in class speech in real time. Combining the theoretical research of information technology-driven intelligent evaluation of classroom teaching, 6 lessons were selected as samples, and the original S-T analysis method and real-time S-T method were used for comparative study, which verified the effectiveness of the proposed real-time S-T analysis method.

Real Time Speech To Text Text To Speech Converter With Automatic Text Summarizer using Natural Language Generation And Abstract Meaning Representation

Due to extensive needs for growth in various sectors, which include software, telecom, healthcare, defence, etc., there is a necessary increase in the number as well as the duration of meetings, conference calls, reconnaissance stakeouts, financial reviews. The obtained reports of these play a significant role in defining the plan of actions. The proposed model is to convert real-time speech to corresponding text and then to its respective summary using Natural Language Grammar (NLG) and Abstract Meaning Representation (AMR) graphs and then again turned back the obtained summary to speech. The proposed model intends to achieve the task using two major algorithms, 1) Deep Speech 2, 2) AMR graphs. The speech-recognition model recommended has a speedup of 4x if the algorithm runs on a Central Processing Unit (CPU), and the use of particular Graphics Processing Units (GPUs) for running deep learning algorithms can give a speedup of 21x. The performance of the summarizer used is close to the Lead-3-AMR-Baseline model, which is a solid baseline for the CNN/Dailymail dataset. The summarizer we use scores ROGUE score close to the Lead-3- AMR-Baseline model with an accuracy of 99.37%.

EMSContExt: EMS Protocol-Driven Concept Extraction for Cognitive Assistance in Emergency Response

This paper presents a technique for automated curation of a domain-specific knowledge base or lexicon for resource-constrained domains, such as Emergency Medical Services (EMS) and its application to real-time concept extraction and cognitive assistance in emergency response. The EMS responders often verbalize critical information describing the situations at an incident scene, including patients' physical condition and medical history. Automated extraction of EMS protocol-specific concepts from responders' speech data can facilitate cognitive support through the selection and execution of the proper EMS protocols for patient treatment. Although this task is similar to the traditional NLP task of concept extraction, the underlying application domain poses major challenges, including low training resources availability (e.g., no existing EMS ontology, lexicon, or annotated EMS corpus) and domain mismatch. Hence, we develop EMSContExt, a weakly-supervised concept extraction approach for EMS concepts. It utilizes different knowledge bases and a semantic concept model based on a corpus of over 9400 EMS narratives for lexicon expansion. The expanded EMS lexicon is then used to automatically extract critical EMS protocol-specific concepts from real-time EMS speech narratives. Our experimental results show that EMSContExt achieves 0.85 recall and 0.82 F1-score for EMS concept extraction and significantly outperforms MetaMap, a state-of-the-art medical concept extraction tool. We also demonstrate the application of EMSContExt to EMS protocol selection and execution and real-time recommendation of protocol-specific interventions to the EMS responders. Here, EMSContExt outperforms MetaMap with a 6% increase and six times speedup in weighted recall and execution time, respectively.

Multiclass covert speech classification using extreme learning machine.

The objective of the proposed research is to classify electroencephalography (EEG) data of covert speech words. Six subjects were asked to perform covert speech tasks i.e mental repetition of four different words i.e 'left', 'right', 'up' and 'down'. Fifty trials for each word recorded for every subject. Kernel-based Extreme Learning Machine (kernel ELM) was used for multiclass and binary classification of EEG signals of covert speech words. We achieved a maximum multiclass and binary classification accuracy of (49.77%) and (85.57%) respectively. The kernel ELM achieves significantly higher accuracy compared to some of the most commonly used classification algorithms in Brain-Computer Interfaces (BCIs). Our findings suggested that covert speech EEG signals could be successfully classified using kernel ELM. This research involving the classification of covert speech words potentially leading to real-time silent speech BCI research.

RETRACTED: Real time speech recognition algorithm on embedded system based on continuous Markov model

Abstract Real time speech recognition technology, as a key cross technology in the field of artificial intelligence in recent years, has been widely used in the fields of intelligent voice toys, industrial control and intelligent rehabilitation. Because the real-time speech recognition technology based on embedded technology has obvious advantages in the volume, power consumption and research and development cost of the system, it has become a hot carrier to achieve efficient speech recognition technology. In order to realize a simple and practical real-time speech recognition system based on embedded system, this paper designs a basic framework of machine learning based on Markov random field theory combined with machine learning theory, and studies the algorithm of real-time speech vocabulary matching recognition based on this framework. In detail, the algorithm proposed in this paper will process the speech signal from the aspects of preprocessing, signal detection, feature extraction and quantization. Finally, this paper will build a real-time speech recognition system based on DSP processor. The experimental results show that the real-time speech recognition algorithm proposed in this paper can improve the real-time recognition speed of the system. The corresponding speed changes from about 12 s to about 200 ms, and the corresponding real-time accuracy rate increases to about 95%.

The Pransky interview: Dr James Kuffner, CEO at Toyota Research Institute Advanced Development, Coinventor of the rapidly, exploring random tree algorithm

PurposeThe following article is a “Q&A interview” conducted by Joanne Pransky of Industrial Robot Journal as a method to impart the combined technological, business and personal experience of a prominent, robotic industry PhD-turned entrepreneur regarding his pioneering efforts of bringing technological inventions to market. The paper aims to discuss these issues.Design/methodology/approachThe interviewee is Dr James Kuffner, CEO at Toyota Research Institute Advanced Development (TRI-AD). Kuffner is a proven entrepreneur and inventor in robot and motion planning and cloud robotics. In this interview, Kuffner shares his personal and professional journey from conceptualization to commercial realization.FindingsDr Kuffner received BS, MS and PhD degrees from the Stanford University’s Department of Computer Science Robotics Laboratory. He was a Japan Society for the Promotion of Science (JSPS) Postdoctoral Research Fellow at the University of Tokyo where he worked on software and planning algorithms for humanoid robots. He joined the faculty at Carnegie Mellon University’s Robotics Institute in 2002 where he served until March 2018. Kuffner was a Research Scientist and Engineering Director at Google from 2009 to 2016. In January 2016, he joined TRI where he was appointed the Chief Technology Officer and Area Lead, Cloud Intelligence and is presently an Executive Advisor. He has been CEO of TRI-AD since April of 2018.Originality/valueDr Kuffner is perhaps best known as the co-inventor of the rapidly exploring random tree (RRT) algorithm, which has become a key standard benchmark for robot motion planning. He is also known for introducing the term “Cloud Robotics” in 2010 to describe how network-connected robots could take advantage of distributed computation and data stored in the cloud. Kuffner was part of the initial engineering team that built Google’s self-driving car. He was appointed Head of Google’s Robotics Division in 2014, which he co-founded with Andy Rubin to help realize the original Cloud Robotics concept. Kuffner also co-founded Motion Factory, where he was the Senior Software Engineer and a member of the engineering team to develop C++ based authoring tools for high-level graphic animation and interactive multimedia content. Motion Factory was acquired by SoftImage in 2000. In May 2007, Kuffner founded, and became the Director of Robot Autonomy where he coordinated research and software consulting for industrial and consumer robotics applications. In 2008, he assisted in the iOS development of Jibbigo, the first on-phone, real-time speech recognition, translation and speech synthesis application for the iPhone. Jibbigo was acquired by Facebook in 2013. Kuffner is one of the most highly cited authors in the field of robotics and motion planning, with over 15,000 citations. He has published over 125 technical papers and was issued more than 50 patents related to robotics and computer vision technology.