Speech Commands Dataset Research Articles

Application of Deep Learning for Voice Command Classification in Turkish Language

In this study, a deep learning model was developed for the recognition and classification of voice commands using the Turkish Speech Command Dataset. The division of training, validation, and test sets was carried out on an individual basis. This approach aims to prevent the model from memorizing and to enhance its generalization capability. The model was trained using Mel-Frequency Cepstral Coefficients (MFCC) features extracted from voice files, and its classification performance was evaluated in detail. The findings indicate that the model successfully classifies voice commands with a high accuracy rate, achieving an overall accuracy of 92.3% on the test set, highlighting the potential of deep learning approaches in voice recognition technologies

CNN Based Automatic Speech Recognition: A Comparative Study

Recently, one of the most common approaches used in speech recognition is deep learning. The most advanced results have been obtained with speech recognition systems created using convolutional neural network (CNN) and recurrent neural networks (RNN). Since CNNs can capture local features effectively, they are applied to tasks with relatively short-term dependencies, such as keyword detection or phoneme- level sequence recognition. This paper presents the development of a deep learning and speech command recognition system. The Google Speech Commands Dataset has been used for training. The dataset contained 65.000 one-second-long words of 30 short English words. That is, %80 of the dataset has been used in the training and %20 of the dataset has been used in the testing. The data set consists of one-second voice commands that have been converted into a spectrogram and used to train different artificial neural network (ANN) models. Various variants of CNN are used in deep learning applications. The performance of the proposed model has reached %94.60.

9.1 µW keyword spotting processor based on optimized MFCC and small‐footprint TENet in 28‐nm CMOS

AbstractThis letter proposes a low‐power keyword spotting (KWS) architecture based on a modified temporal efficient neural network (TENet) and a simplified mel‐frequency cepstrum coefficient (MFCC) algorithm. The optimized MFCC algorithm reduces the computational load by 82% for multiplications and 66% for additions. An efficient hardware architecture and data flow for TENet have been designed, resulting in a 3.1× reduction in the operating cycle compared to similar work. The parameter count and computational load are reduced by 3.7× and 2.8×, respectively, and the accuracy reaches 95.36% for ten keywords in the Google Speech Command Dataset (GSCD). Operating at a frequency of 16 KHz for MFCC and 100 KHz for NN accelerator on a 28 nm process, the power consumption overhead is 9.1 µW.

Co-learning synaptic delays, weights and adaptation in spiking neural networks.

Spiking neural network (SNN) distinguish themselves from artificial neural network (ANN) because of their inherent temporal processing and spike-based computations, enabling a power-efficient implementation in neuromorphic hardware. In this study, we demonstrate that data processing with spiking neurons can be enhanced by co-learning the synaptic weights with two other biologically inspired neuronal features: (1) a set of parameters describing neuronal adaptation processes and (2) synaptic propagation delays. The former allows a spiking neuron to learn how to specifically react to incoming spikes based on its past. The trained adaptation parameters result in neuronal heterogeneity, which leads to a greater variety in available spike patterns and is also found in the brain. The latter enables to learn to explicitly correlate spike trains that are temporally distanced. Synaptic delays reflect the time an action potential requires to travel from one neuron to another. We show that each of the co-learned features separately leads to an improvement over the baseline SNN and that the combination of both leads to state-of-the-art SNN results on all speech recognition datasets investigated with a simple 2-hidden layer feed-forward network. Our SNN outperforms the benchmark ANN on the neuromorphic datasets (Spiking Heidelberg Digits and Spiking Speech Commands), even with fewer trainable parameters. On the 35-class Google Speech Commands dataset, our SNN also outperforms a GRU of similar size. Our study presents brain-inspired improvements in SNN that enable them to excel over an equivalent ANN of similar size on tasks with rich temporal dynamics.

Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training

Recurrent Neural Networks (RNNs) are useful in temporal sequence tasks. However, training RNNs involves dense matrix multiplications which require hardware that can support a large number of arithmetic operations and memory accesses. Implementing online training of RNNs on the edge calls for optimized algorithms for an efficient deployment on hardware. Inspired by the spiking neuron model, the Delta RNN exploits temporal sparsity during inference by skipping over the update of hidden states from those inactivated neurons whose change of activation across two timesteps is below a defined threshold. This work describes a training algorithm for Delta RNNs that exploits temporal sparsity in the backward propagation phase to reduce computational requirements for training on the edge. Due to the symmetric computation graphs of forward and backward propagation during training, the gradient computation of inactivated neurons can be skipped. Results show a reduction of ∼80% in matrix operations for training a 56k parameter Delta LSTM on the Fluent Speech Commands dataset with negligible accuracy loss. Logic simulations of a hardware accelerator designed for the training algorithm show 2-10X speedup in matrix computations for an activation sparsity range of 50%-90%. Additionally, we show that the proposed Delta RNN training will be useful for online incremental learning on edge devices with limited computing resources.

AUDIO SPECTROGRAM TRANSFORMER (AST): ADVANTAGES OVER TRADITIONAL ALGORITHMS IN SPEECH-TO-TEXT (STT)

Automatic Speech Recognition (ASR) has seen significant advancements in recent years, largely due to the development of deep learning models. One of the most notable advancements is the Spectrogram Transformer, a variant of the Transformer architecture tailored for audio processing tasks. In this paper, we review the Spectrogram Transformer and compare it with other traditional ASR algorithms. We discuss its benefits, such as improved performance on noisy audio and better modeling of long-range dependencies. Additionally, we explore its applications in various domains, including voice assistants, transcription services, and audio indexing. Through experiments on benchmark datasets like LibriSpeech and the Speech Commands Dataset, we demonstrate the effectiveness of the Spectrogram Transformer in achieving state-of-the-art performance. Our findings suggest that the Spectrogram Transformer offers a promising direction for future advancements in ASR technology.

Bidirectional Representations for Low-Resource Spoken Language Understanding

Speech representation models lack the ability to efficiently store semantic information and require fine tuning to deliver decent performance. In this research, we introduce a transformer encoder–decoder framework with a multiobjective training strategy, incorporating connectionist temporal classification (CTC) and masked language modeling (MLM) objectives. This approach enables the model to learn contextual bidirectional representations. We evaluate the representations in a challenging low-resource scenario, where training data is limited, necessitating expressive speech embeddings to compensate for the scarcity of examples. Notably, we demonstrate that our model’s initial embeddings outperform comparable models on multiple datasets before fine tuning. Fine tuning the top layers of the representation model further enhances performance, particularly on the Fluent Speech Command dataset, even under low-resource conditions. Additionally, we introduce the concept of class attention as an efficient module for spoken language understanding, characterized by its speed and minimal parameter requirements. Class attention not only aids in explaining model predictions but also enhances our understanding of the underlying decision-making processes. Our experiments cover both English and Dutch languages, offering a comprehensive evaluation of our proposed approach.

The Power-Normalized Cepstral Coefficient (PNCC) for convolutional neural networks-based robust speech command recognition

While implementations of speech recognition grow rapidly in recent years and are slowly being integrated into our daily devices, the problem of noise robustness is still a challenging task, even with the recent advancement of deep learning technologies for speech recognition. The presence of noise may cause a mismatch between training, which is performed in clean conditions, and noisy testing conditions. This paper proposes a method to extract features for speech recognition by employing features derived under the power law scale, i.e., the Power-Normalized Cepstral Coefficient (PNCC). The power-law can provide better compression in low-energy regions so that it is not sensitive when the speech signal is distorted by noise. The features are implemented on speech recognition based on Convolutional Neural Networks (CNNs). The experiments were carried out by TensorFlow’s Speech Command Dataset mixed with various signal-to-noise ratio to evaluate the method. The experimental findings indicate that the accuracy ranges from 81% to 86%.

ATOSE: Audio Tagging with One-Sided Joint Embedding

Audio auto-tagging is the process of assigning labels to audio clips for better categorization and management of audio file databases. With the advent of advanced artificial intelligence technologies, there has been increasing interest in directly using raw audio data as input for deep learning models in order to perform tagging and eliminate the need for preprocessing. Unfortunately, most current studies of audio auto-tagging cannot effectively reflect the semantic relationships between tags—for instance, the connection between “classical music” and “cello”. In this paper, we propose a novel method that can enhance audio auto-tagging performance via joint embedding. Our model has been carefully designed and architected to recognize the semantic information within the tag domains. In our experiments using the MagnaTagATune (MTAT) dataset, which has high inter-tag correlations, and the Speech Commands dataset, which has no inter-tag correlations, we showed that our approach improves the performance of existing models when there are strong inter-tag correlations.

Efficient Self-Attention Model for Speech Recognition-Based Assistive Robots Control.

Assistive robots are tools that people living with upper body disabilities can leverage to autonomously perform Activities of Daily Living (ADL). Unfortunately, conventional control methods still rely on low-dimensional, easy-to-implement interfaces such as joysticks that tend to be unintuitive and cumbersome to use. In contrast, vocal commands may represent a viable and intuitive alternative. This work represents an important step toward providing a viable vocal interface for people living with upper limb disabilities by proposing a novel lightweight vocal command recognition system. The proposed model leverages the MobileNet2 architecture, augmenting it with a novel approach to the self-attention mechanism, achieving a new state-of-the-art performance for Keyword Spotting (KWS) on the Google Speech Commands Dataset (GSCD). Moreover, this work presents a new dataset, referred to as the French Speech Commands Dataset (FSCD), comprising 4963 vocal command utterances. Using the GSCD as the source, we used Transfer Learning (TL) to adapt the model to this cross-language task. TL has been shown to significantly improve the model performance on the FSCD. The viability of the proposed approach is further demonstrated through real-life control of a robotic arm by four healthy participants using both the proposed vocal interface and a joystick.

Recognizing Command Words using Deep Recurrent Neural Network for Both Acoustic and Throat Speech

The importance of speech command recognition in a human-machine interaction system is increased in recent years. In this study, we propose a deep neural network-based system for acoustic and throat command speech recognition. We apply a preprocessed pipeline to create the input of the deep learning model. Firstly, speech commands are decomposed into components using well-known signal decomposition techniques. The Mel-frequency cepstral coefficients (MFCC) feature extraction method is applied to each component of the speech commands to obtain the feature inputs for the recognition system. At this stage, we apply and compare performance using different speech decomposition techniques such as wavelet packet decomposition (WPD), continuous wavelet transform (CWT), and empirical mode decomposition (EMD) in order to find out the best technique for our model. We observe that WPD shows the best performance in terms of classification accuracy. This paper investigates long short-term memory (LSTM)-based recurrent neural network (RNN), which is trained using the extracted MFCC features. The proposed neural network is trained and tested using acoustic speech commands. Moreover, we also train and test the proposed model using a throat mic. speech commands as well. Lastly, the transfer learning technique is employed to increase the test accuracy for throat speech recognition. The weights of the model train with the acoustic signal are used to initialize the model used for throat speech recognition. Overall, we have found significant classification accuracy for both acoustic and throat command speech. We obtain LSTM is much better than the GMM-HMM model, convolutional neural networks such as CNN-tpool2 and residual networks such as res15 and res26 with an accuracy score of over 97% on Google’s Speech Commands dataset and we achieve 95.35% accuracy on our throat speech data set using the transfer learning technique.

Broadcast Attention Learning for Real Telephone Speech Keyword Spotting

With the development of mobile smart devices, keyword spotting plays an important role in the interaction between machines and users. However, low storage and low energy consumption of mobile devices limit the accuracies of keyword spotting tasks. Therefore, how to achieve a balance between the high accuracy and low consumption is a research hotspot for a keyword spotting system. Convolutional neural networks have been widely adopted in recent keyword spotting systems due to their superior accuracies, and the success of the transformer architecture in many areas demonstrates the effectiveness of self-attention. In this paper, we combine self-attention and convolutional neural networks, and propose the broadcast attention learning network (BA-net), using a small number of parameters while achieving the accuracies of 97.18% and 78.44% respectively on the Google speech command dataset and a real telephone speech dataset.

Memristor-Based Light-Weight Transformer Circuit Implementation for Speech Recognizing

Transformer network (TN) is a promising model widely used for natural language processing (NLP), computer vision (CV), and audio processing (AP). However, the large number of multiples and additions (multi-adds) in TN leads to a massive number of parameters in the TN model, causing an increasing number of hardware resource overhead and thus making it difficult to deploy the TN model to edge devices. To alleviate this problem, this paper proposes a circuit implementation scheme for a memristor-based lightweight transformer network (MLTN). In addition, the speech recognition task based on the speech command dataset is implemented through the modular hardware design approach, which provides an effective lightweight hardware implementation scheme for the edge devices. Further, the retention, noise immunity, variability, and stability of the designed circuit are analyzed by simulation experiments with LTSPICE. The results show that the proposed MLTN circuit not only has a lighter network model but also has significant recognition accuracy with different hardware scales. Specifically, MLTN-40 can achieve 96.18% simulation accuracy, while the energy overhead is only 3.8% of the ARM framework, which is commonly used for edge devices, and 39.9% of the traditional server (GPU), making it a promising solution for implementing speech recognition on edge devices.

AAD-KWS: A Sub-μ W Keyword Spotting Chip With an Acoustic Activity Detector Embedded in MFCC and a Tunable Detection Window in 28-nm CMOS

As a widely used speech-triggered interface, deep-learning-based keyword spotting (KWS) chips require both ultra-low power and high detection accuracy. We propose a sub-microwatt KWS chip with an acoustic activity detection (AAD) to achieve the above two requirements, including the following techniques: first, an optimized feature extractor circuit using nonoverlapping-framed serial Mel frequency cepstral coefficient (MFCC) to save half of the computations and data storage; second, a zero-cost AAD by using MFCC’s 1st-order output to clock gate neural network (NN) and postprocessing (PP) unit, with 0 miss rate; third, a tunable detection window to adapt to different keyword lengths for better accuracy; and finally, a true form computation method to decrease data transitions and optimized PP. Implemented in a 28-nm CMOS process, this AAD-KWS chip has a 0.4-V supply, an 8-kHz frequency for MFCC, and a 200-kHz frequency for other parts. It consumes <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.36~\mu \text{W}$ </tex-math></inline-formula> in quiet scenarios when AAD is enabled and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.8~\mu \text{W}$ </tex-math></inline-formula> in normal scenarios, where the MFCC circuit consumes only 170 nW. Its accuracy reaches 97.8% for two keywords in the Google Speech Command Dataset (GSCD).

Image approach to English digits recognition using deep learning

Despite good progress in speech recognition, various challenges still exist due to differences in how they speak, age, gender, emotions, and dialects when perceived by the ear. There is a proverb “I hear, and I forget; I see, and I remember”. The image would be another solution to recognize what we hear. The main objective of this paper is to investigate the graphic method to learn digit English using the Deep Learning technique. In this work, Mel-frequency cepstral coefficients (MFCC) in the form of an image will be used as input to the system. Convolutional neural network (CNN) will be used to extract features from the image and an artificial neural network (ANN) will be used to classify those features into 10-digit English classes. By using the Speech Command dataset, the performance of the system will be compared with a conventional method that uses MFCC features in the form of a signal. The experiments showed that the image approach improves the recognition rate from 49% to 84%. It can be concluded that image approach can be used as an alternative method for digit recognition.

Evaluating Robustness to Noise and Compression of Deep Neural Networks for Keyword Spotting

Keyword Spotting (KWS) has been the subject of research in recent years given the increase of embedded systems for command recognition such as Alexa, Google Home, and Siri. Performance, model size, processing time, and robustness to noise are fundamental in these systems. Furthermore, applications in embedded systems demand computationally efficient models that can be implemented in current technology. In this work, an approach for keyword recognition is evaluated using three deep learning models namely LeNet-5, SqueezeNet, and EfficientNet-B0. We evaluate transfer learning, pruning and quantization strategies in training and test using noisy and clean speech signals. In addition, compression techniques such as pruning and quantization were assessed in terms of the size reduction of the model footprint and the accuracy obtained in each case. Using the Google’s Speech Commands dataset and additive babble noise signal, our keyword recognition approach achieves an accuracy of 94.6% using an unstructured pruning of 80% of the parameters of the original SqueezeNet network with a reduction of 70% in the model size.

E-prop on SpiNNaker 2: Exploring online learning in spiking RNNs on neuromorphic hardware.

In recent years, the application of deep learning models at the edge has gained attention. Typically, artificial neural networks (ANNs) are trained on graphics processing units (GPUs) and optimized for efficient execution on edge devices. Training ANNs directly at the edge is the next step with many applications such as the adaptation of models to specific situations like changes in environmental settings or optimization for individuals, e.g., optimization for speakers for speech processing. Also, local training can preserve privacy. Over the last few years, many algorithms have been developed to reduce memory footprint and computation. A specific challenge to train recurrent neural networks (RNNs) for processing sequential data is the need for the Back Propagation Through Time (BPTT) algorithm to store the network state of all time steps. This limitation is resolved by the biologically-inspired E-prop approach for training Spiking Recurrent Neural Networks (SRNNs). We implement the E-prop algorithm on a prototype of the SpiNNaker 2 neuromorphic system. A parallelization strategy is developed to split and train networks on the ARM cores of SpiNNaker 2 to make efficient use of both memory and compute resources. We trained an SRNN from scratch on SpiNNaker 2 in real-time on the Google Speech Command dataset for keyword spotting. We achieved an accuracy of 91.12% while requiring only 680 KB of memory for training the network with 25 K weights. Compared to other spiking neural networks with equal or better accuracy, our work is significantly more memory-efficient. In addition, we performed a memory and time profiling of the E-prop algorithm. This is used on the one hand to discuss whether E-prop or BPTT is better suited for training a model at the edge and on the other hand to explore architecture modifications to SpiNNaker 2 to speed up online learning. Finally, energy estimations predict that the SRNN can be trained on SpiNNaker2 with 12 times less energy than using a NVIDIA V100 GPU.

A 108-nW 0.8-mm2 Analog Voice Activity Detector Featuring a Time-Domain CNN With Sparsity-Aware Computation and Sparsified Quantization in 28-nm CMOS

This article reports a passive analog feature extractor for realizing an area-and-power-efficient voice activity detector (VAD) for voice-control edge devices. It features a switched-capacitor circuit as the time-domain convolutional neural network (TD-CNN) that extracts the 1-bit features for the subsequent binarized neural network (BNN) classifier. TD-CNN also allows area savings and low latency by evaluating the features temporally. The applied sparsity-aware computation (SAC) and sparsified quantization (SQ) aid in enlarging the output swing and reducing the model size without sacrificing the classification accuracy. With these techniques, the diversified output also aids in desensitizing the 1-bit quantizer from the offset and noise. The TD-CNN and BNN are trained as a single network to improve the VAD reconfigurability. Benchmarking with the prior art, our VAD in 28-nm CMOS scores a 90% (94%) speech (non-speech) hit rate on the TIMIT dataset with small power (108 nW) and area (0.8 mm2). We can configure the TD-CNN as a feature extractor for keyword spotting (KWS). It achieves a 93.5% KWS accuracy with the Google speech command dataset (two keywords). With two TD-CNNs operating simultaneously to extract more features, the KWS accuracy is 94.3%.

A Configurable Accelerator for Keyword Spotting Based on Small-Footprint Temporal Efficient Neural Network

Keyword spotting (KWS) plays a crucial role in human–machine interactions involving smart devices. In recent years, temporal convolutional networks (TCNs) have performed outstandingly with less computational complexity, in comparison with classical convolutional neural network (CNN) methods. However, it remains challenging to achieve a trade-off between a small-footprint model and high accuracy for the edge deployment of the KWS system. In this article, we propose a small-footprint model based on a modified temporal efficient neural network (TENet) and a simplified mel-frequency cepstrum coefficient (MFCC) algorithm. With the batch-norm folding and int8 quantization of the network, our model achieves the accuracy of 95.36% on Google Speech Command Dataset (GSCD) with only 18 K parameters and 461 K multiplications. Furthermore, following a hardware/model co-design approach, we propose an optimized dataflow and a configurable hardware architecture for TENet inference. The proposed accelerator implemented on Xilinx zynq 7z020 achieves an energy efficiency of 25.6 GOPS/W and reduces the runtime by 3.1× compared with state-of-the-art work.

Quality Driven Systematic Approximation for Binary-Weight Neural Network Deployment

Neural networks (NNs) with large scales of artificial neurons are increasingly used in recognition and classification tasks. In power-constrained scenarios, the tradeoff between performance and hardware consumptions must be carefully evaluated before silicon tape-out. In this paper, we proposed a systematic approach to design ultra-low power NN system. This work is motivated by the facts that NNs are resilient to approximation in many of the computations and NNs are outputting statistical tensors which are acceptable to less-than-perfect results. We resort to the front-back end approach with a twofold aim: (1) a fast and accurate design approach is proposed by estimating the computing quality of low-power approximate adder arrays, and it is adopted to evaluate the neural network system; (2) a quality configurable engine with different approximation degrees while processing NNs is implemented. The proposed work is demonstrated with a comprehensive keyword spotting (KWS) system as an ultra-low power NN engine. The experimental environment is setup with ten keywords from the google speech command dataset (GSCD) using an industrial 22-nm ultra-low-leakage (ULL) process. Comparing to the state-of-the-art KWS processors, the proposed approximate NN engine can demonstrate over 60% improvement in power efficiency and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.1\times $ </tex-math></inline-formula> area efficiency while achieving similar recognition accuracy.