Keyword Spotting System Research Articles

Artificial basilar membrane/hair cell integrated acoustic system for keyword spotting in noisy environments inspired by human cochlea

We report a novel speech recognition method using a noise-robust acoustic sensor system that integrates a spatially frequency-separating sensor with a nonlinear amplification algorithm, mimicking the cochlea’s basilar membrane and hair cells. The multichannel piezoelectric artificial basilar membrane (ABM) sensor detects specific sound frequencies with high sensitivity over 0.2–6 kHz. The signal processing model of the Artificial Hair Cell inspired by the signal transduction mechanism of inner hair cells, simultaneously enhances the frequency selectivity of ABM sensors and improves noise robustness. In a 0 dB SNR noisy environment, it effectively detected the voice signal with a maximum SNR of 57 dB. Furthermore, we converted the frequency-separated signals for speech sounds in various noisy environments into heatmap images and utilized them as input for a CNN-based speech recognition algorithm. Consequently, our system demonstrated noise-robust recognition performance with 94 % accuracy, even in noisy environments.

Advantages and Pitfalls of Dataset Condensation: An Approach to Keyword Spotting with Time-Frequency Representations

With the exponential growth of data, the need for efficient techniques to extract relevant information from datasets becomes increasingly imperative. Reducing the training data can be useful for applications wherein storage space or computational resources are limited. In this work, we explore the concept of data condensation (DC) in the context of keyword spotting systems (KWS). Using deep learning architectures and time-frequency speech representations, we have obtained condensed speech signal representations using gradient matching with Efficient Synthetic-Data Parameterization. From a series of classification experiments, we analyze the models and condensed data performances in terms of accuracy and number of data per class. We also present results using cross-model techniques, wherein models are trained with condensed data obtained from a different architecture. Our findings demonstrate the potential of data condensation in the context of the speech domain for reducing the size of datasets while retaining their most important information and maintaining high accuracy for the model trained with the condensed dataset. We have obtained an accuracy of 80.75% with 30 condensed speech representations per class with ConvNet, representing an addition of 24.9% in absolute terms when compared to 30 random samples from the original training dataset. However, we demonstrate the limitations of this approach in the cross-model tests. We also highlight the challenges and opportunities for further improving the accuracy of condensed data obtained and trained with different neural network architectures.

Robust Dual-Modal Speech Keyword Spotting for XR Headsets.

While speech interaction finds widespread utility within the Extended Reality (XR) domain, conventional vocal speech keyword spotting systems continue to grapple with formidable challenges, including suboptimal performance in noisy environments, impracticality in situations requiring silence, and susceptibility to inadvertent activations when others speak nearby. These challenges, however, can potentially be surmounted through the cost-effective fusion of voice and lip movement information. Consequently, we propose a novel vocal-echoic dual-modal keyword spotting system designed for XR headsets. We devise two different modal fusion approches and conduct experiments to test the system's performance across diverse scenarios. The results show that our dual-modal system not only consistently outperforms its single-modal counterparts, demonstrating higher precision in both typical and noisy environments, but also excels in accurately identifying silent utterances. Furthermore, we have successfully applied the system in real-time demonstrations, achieving promising results. The code is available at https://github.com/caizhuojiang/VE-KWS.

Hardware–Software Co-Design of an Audio Feature Extraction Pipeline for Machine Learning Applications

Keyword spotting is an important part of modern speech recognition pipelines. Typical contemporary keyword-spotting systems are based on Mel-Frequency Cepstral Coefficient (MFCC) audio features, which are relatively complex to compute. Considering the always-on nature of many keyword-spotting systems, it is prudent to optimize this part of the detection pipeline. We explore the simplifications of the MFCC audio features and derive a simplified version that can be more easily used in embedded applications. Additionally, we implement a hardware generator that generates an appropriate hardware pipeline for the simplified audio feature extraction. Using Chisel4ml framework, we integrate hardware generators into Python-based Keras framework, which facilitates the training process of the machine learning models using our simplified audio features.

Design Of Emergency Keyword Recognition Using Arduino Nano BLE Sense 33 And Edge Impulse

This project focuses on Custom Keyword Voice Recognition (CKVR) for emergency response scenarios. A multilingual keyword spotting system is developed using the Arduino Nano 33 BLE Sense board and Edge Impulse. The system accurately recognizes the keyword "help" in English, Arabic, Kurdish, and Malay languages. The project utilizes Mel Frequency Cepstral Coefficients (MFCC) for feature extraction and employs deep learning techniques for model training. By optimizing the model through quantization and achieving 100% accuracy in training and testing phases, the system provides a reliable solution for identifying emergency keywords. The developed system has the potential to enhance safety in public spaces such as malls, hospitals, schools, and stations by quickly responding to individuals in distress. The project demonstrates the effectiveness of the chosen approach, highlighting the significance of MFCC processing, classification learning, and optimized model design in speech recognition. The successful development of this keyword spotting system opens doors for further advancements in the field and emphasizes the potential for innovative solutions that contribute to a safer and more responsive world.

A Low-Power Keyword Spotting System With High-Order Passive Switched-Capacitor Bandpass Filters for Analog-MFCC Feature Extraction

A Low-Power Keyword Spotting System With High-Order Passive Switched-Capacitor Bandpass Filters for Analog-MFCC Feature Extraction

A Resource-Efficient Keyword Spotting System Based on a One-Dimensional Binary Convolutional Neural Network

This paper proposes a resource-efficient keyword spotting (KWS) system based on a convolutional neural network (CNN). The end-to-end KWS process is performed based solely on 1D-CNN inference, where features are first extracted from a few convolutional blocks, and then the keywords are classified using a few fully connected blocks. The 1D-CNN model is binarized to reduce resource usage, and its inference is executed by employing a dedicated engine. This engine is designed to skip redundant operations, enabling high inference speed despite its low complexity. The proposed system is implemented using 6895 ALUTs in an Intel Cyclone V FPGA by integrating the essential components for performing the KWS process. In the system, the latency required to process a frame is 22 ms, and the spotting accuracy is 91.80% in an environment where the signal-to-noise ratio is 10 dB for Google speech commands dataset version 2.

A 34.7 µW Speech Keyword Spotting IC Based on Subband Energy Feature Extraction

In the era of the Internet of Things (IoT), voice control has enhanced human–machine interaction and the accuracy of keyword spotting (KWS) algorithms has reached 97%; however, the high power consumption of KWS algorithms caused by their huge computing and storage requirements has limited their application in Artificial Intelligence of Things (AIoT) devices. In this study, voice features are extracted by utilizing the fast discrete cosine transform (FDCT) for frequency-domain transformation and to shorten the process of calculating the logarithmic spectrum and cepstrum. The designed KWS system is a two-stage wake-up system, with a sound detection (SD) awakening KWS. The inference process of the KWS network is achieved using time-division computation, reducing the KWS clock to an ultra-low frequency of 24 kHz.At the same time, the implementation of a depthwise separable convolution neural network (DSCNN) greatly reduces the parameter quantity and computation. Under the GSMC 0.11 µm technology, post-layout simulation results show that the total synthesized area of the entire system circuit is 0.58 mm2, the power consumption is 34.7 µW, and the F1-score of the KWS is 0.89 with 10 dB noise, which makes it suitable as a KWS system in AIoT devices.

Multi-task deep cross-attention networks for far-field speaker verification and keyword spotting

Personalized voice triggering is a key technology in voice assistants and serves as the first step for users to activate the voice assistant. Personalized voice triggering involves keyword spotting (KWS) and speaker verification (SV). Conventional approaches to this task include developing KWS and SV systems separately. This paper proposes a single system called the multi-task deep cross-attention network (MTCANet) that simultaneously performs KWS and SV, while effectively utilizing information relevant to both tasks. The proposed framework integrates a KWS sub-network and an SV sub-network to enhance performance in challenging conditions such as noisy environments, short-duration speech, and model generalization. At the core of MTCANet are three modules: a novel deep cross-attention (DCA) module to integrate KWS and SV tasks, a multi-layer stacked shared encoder (SE) to reduce the impact of noise on the recognition rate, and soft attention (SA) modules to allow the model to focus on pertinent information in the middle layer while preventing gradient vanishing. Our proposed model demonstrates outstanding performance in the well-off test set, improving by 0.2%, 0.023, and 2.28% over the well-known SV model emphasized channel attention, propagation, and aggregation in time delay neural network (ECAPA-TDNN) and the advanced KWS model Convmixer in terms of equal error rate (EER), minimum detection cost function (minDCF), and accuracy (Acc), respectively.

FPGA Implementation of Keyword Spotting System Using Depthwise Separable Binarized and Ternarized Neural Networks.

Keyword spotting (KWS) systems are used for human-machine communications in various applications. In many cases, KWS involves a combination of wake-up-word (WUW) recognition for device activation and voice command classification tasks. These tasks present a challenge for embedded systems due to the complexity of deep learning algorithms and the need for optimized networks for each application. In this paper, we propose a depthwise separable binarized/ternarized neural network (DS-BTNN) hardware accelerator capable of performing both WUW recognition and command classification on a single device. The design achieves significant area efficiency by redundantly utilizing bitwise operators in the computation of the binarized neural network (BNN) and ternary neural network (TNN). In a complementary metal-oxide semiconductor (CMOS) 40 nm process environment, the DS-BTNN accelerator demonstrated significant efficiency. Compared with a design approach where BNN and TNN were independently developed and subsequently integrated as two separate modules into the system, our method achieved a 49.3% area reduction while yielding an area of 0.558 mm2. The designed KWS system, which was implemented on a Xilinx UltraScale+ ZCU104 field-programmable gate array (FPGA) board, receives real-time data from the microphone, preprocesses them into a mel spectrogram, and uses this as input to the classifier. Depending on the order, the network operates as a BNN or a TNN for WUW recognition and command classification, respectively. Operating at 170 MHz, our system achieved 97.1% accuracy in BNN-based WUW recognition and 90.5% in TNN-based command classification.

Broadcast Attention Learning for Real Telephone Speech Keyword Spotting

With the development of mobile smart devices, keyword spotting plays an important role in the interaction between machines and users. However, low storage and low energy consumption of mobile devices limit the accuracies of keyword spotting tasks. Therefore, how to achieve a balance between the high accuracy and low consumption is a research hotspot for a keyword spotting system. Convolutional neural networks have been widely adopted in recent keyword spotting systems due to their superior accuracies, and the success of the transformer architecture in many areas demonstrates the effectiveness of self-attention. In this paper, we combine self-attention and convolutional neural networks, and propose the broadcast attention learning network (BA-net), using a small number of parameters while achieving the accuracies of 97.18% and 78.44% respectively on the Google speech command dataset and a real telephone speech dataset.

KEYWORD SPOTTING SYSTEM WITH NANO 33 BLE SENSE USING EMBEDDED MACHINE LEARNING APPROACH

Due to the obvious advancement of artificial intelligence, keyword spotting has become a fast-growing technology that was first launched a few years ago by hidden Markov models. Keyword spotting is the technique of finding terms that have been pre-programmed into a machine learning model. However, because the keyword spotting system model will be installed on a small and resource-constrained device, it must be minimal in size. It is difficult to maintain accuracy and performance when minimizing the model size. We suggested in this paper to develop a TinyML model that responds to voice commands by detecting words that are utilized in a cascade architecture to start or control a program. The keyword detection machine learning model was built, trained, and tested using the edge impulse development platform. The technique follows the model-building workflow, which includes data collection, preprocessing, training, testing, and deployment. 'On,' 'Off,' noise, and unknown databases were obtained from the Google speech command database V1 and applied for training and testing. The MFCC was used to extract features and CNN was used to generate the model, which was then optimized and deployed on the microcontroller. The model's evaluation represents an accuracy of 84.51% based on the datasets. Finally, the KWS was successfully implemented and assessed on Arduino Nano 33 BLE Sense for two studies in terms of accuracy at three different times and by six different persons.

Fearless Steps APOLLO: Challenges in keyword spotting and topic detection for naturalistic audio streams

Fearless Steps (FS) APOLLO is a + 50,000 hr audio resource established by CRSS-UTDallas capturing all communications between NASA-MCC personnel, backroom staff, and Astronauts across manned Apollo Missions. Such a massive audio resource without metadata/unlabeled corpus provides limited benefit for communities outside Speech-and-Language Technology (SLT). Supplementing this audio with rich metadata developed using robust automated mechanisms to transcribe and highlight naturalistic communications can facilitate open research opportunities for SLT, speech sciences, education, and historical archival communities. In this study, we focus on customizing keyword spotting (KWS) and topic detection systems as an initial step towards conversational understanding. Extensive research in automatic speech recognition (ASR), speech activity, and speaker diarization using manually transcribed 125 h FS Challenge corpus has demonstrated the need for robust domain-specific model development. A major challenge in training KWS systems and topic detection models is the availability of word-level annotations. Forced alignment schemes evaluated using state-of-the-art ASR show significant degradation in segmentation performance. This study explores challenges in extracting accurate keyword segments using existing sentence-level transcriptions and proposes domain-specific KWS-based solutions to detect conversational topics in audio streams. [Work Sponsored by NSF via Grant No. 2016725 and EU’s Horizon 2021 R&I Program under MSC Grant No. 101062614.]

Enhancement of formant regions in magnitude spectra to develop children’s KWS system in zero resource scenario

The mismatches due to pitch, speaking rate, formant dispersion and ambient noise deteriorate the performance of an automatic keyword spotting (KWS) system. The work presented in this paper aims at reducing the aforementioned mismatches through front-end signal processing. In the proposed approach, the short-term magnitude spectra (ST-MS) are firstly computed with a smaller frameshift and then averaged over the adjacent frames to enhance the formant regions. The formant regions in the ST-MS have a higher magnitude than the nearing frequency regions. Consequently, temporal averaging of ST-MS over adjacent frames suppresses the high-frequency variation due to pitch, and formant dispersion. Furthermore, the formant peaks can be more accurately detected from the temporal averaged ST-MS when compared to detection from the original ST-MS. The Mel frequency cepstral coefficients (MFCC) computed from the temporally averaged magnitude spectra (TAS-MFCC) are pitch robust compared to the MFCC, and MFCC extracted from the reported spectral smoothing approaches employing variational mode decomposition (VMD-MFCC), pitch adaptive cepstral truncation (PACT-MFCC) and single-pole filter (SPS-MFCC). Performance of TA-MFCC feature in mismatched test condition is further improved by appending five logarithmically compressed resonant peaks at least separated by 400 Hz, here this feature is termed as TAS-MFCC-ARP. The spectral peaks mostly represent the formants in ST-MS. The performances of the deep neural network-hidden Markov model-based children’s KWS system reported in this work show that the TAS-MFCC-ARP provides a relative performance improvement of 103.83% compared to MFCC. The performance of the KWS system is further improved by data-augmented training through duration modification.

A Configurable Accelerator for Keyword Spotting Based on Small-Footprint Temporal Efficient Neural Network

Keyword spotting (KWS) plays a crucial role in human–machine interactions involving smart devices. In recent years, temporal convolutional networks (TCNs) have performed outstandingly with less computational complexity, in comparison with classical convolutional neural network (CNN) methods. However, it remains challenging to achieve a trade-off between a small-footprint model and high accuracy for the edge deployment of the KWS system. In this article, we propose a small-footprint model based on a modified temporal efficient neural network (TENet) and a simplified mel-frequency cepstrum coefficient (MFCC) algorithm. With the batch-norm folding and int8 quantization of the network, our model achieves the accuracy of 95.36% on Google Speech Command Dataset (GSCD) with only 18 K parameters and 461 K multiplications. Furthermore, following a hardware/model co-design approach, we propose an optimized dataflow and a configurable hardware architecture for TENet inference. The proposed accelerator implemented on Xilinx zynq 7z020 achieves an energy efficiency of 25.6 GOPS/W and reduces the runtime by 3.1× compared with state-of-the-art work.

Generation of Black-box Audio Adversarial Examples Based on Gradient Approximation and Autoencoders

Deep Neural Network (DNN) is gaining popularity thanks to its ability to attain high accuracy and performance in various security-crucial scenarios. However, recent research shows that DNN-based Automatic Speech Recognition (ASR) systems are vulnerable to adversarial attacks. Specifically, these attacks mainly focus on formulating a process of adversarial example generation as iterative, optimization-based attacks. Although these attacks make significant progress, they still take large generation time to produce adversarial examples, which makes them difficult to be launched in real-world scenarios. In this article, we propose a real-time attack framework that utilizes the neural network trained by the gradient approximation method to generate adversarial examples on Keyword Spotting (KWS) systems. The experimental results show that these generated adversarial examples can easily fool a black-box KWS system to output incorrect results with only one inference. In comparison to previous works, our attack can achieve a higher success rate with less than 0.004 s. We also extend our work by presenting a novel ensemble audio adversarial attack and testing the attack on KWS systems equipped with existing defense mechanisms. The efficacy of the proposed attack is well supported by promising experimental results.

Quality Driven Systematic Approximation for Binary-Weight Neural Network Deployment

Neural networks (NNs) with large scales of artificial neurons are increasingly used in recognition and classification tasks. In power-constrained scenarios, the tradeoff between performance and hardware consumptions must be carefully evaluated before silicon tape-out. In this paper, we proposed a systematic approach to design ultra-low power NN system. This work is motivated by the facts that NNs are resilient to approximation in many of the computations and NNs are outputting statistical tensors which are acceptable to less-than-perfect results. We resort to the front-back end approach with a twofold aim: (1) a fast and accurate design approach is proposed by estimating the computing quality of low-power approximate adder arrays, and it is adopted to evaluate the neural network system; (2) a quality configurable engine with different approximation degrees while processing NNs is implemented. The proposed work is demonstrated with a comprehensive keyword spotting (KWS) system as an ultra-low power NN engine. The experimental environment is setup with ten keywords from the google speech command dataset (GSCD) using an industrial 22-nm ultra-low-leakage (ULL) process. Comparing to the state-of-the-art KWS processors, the proposed approximate NN engine can demonstrate over 60% improvement in power efficiency and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.1\times $ </tex-math></inline-formula> area efficiency while achieving similar recognition accuracy.

A Fast Precision Tuning Solution for Always-On DNN Accelerators

Due to the non-volatility nature of Resistive RAM (ReRAM), dynamic operations in the arrays contribute to a much larger portion of power in ReRAM based neural networks than static power. To reduce the dynamic power of in-situ operations with neural parameters, precisiontuning is considered a viable approach of approximate computing to trade-off excessive computation exactness for power and efficiency gains. However, the switching overhead of precision tuning in hardware severely impacts its effectiveness when the systems need to quickly react to the change of environment, user constraint or input quality. This work for the first time investigates the feasibility of agile precision tuning for neural network accelerators to benefit from approximate computing. The proposed Computing in Memory (CiM) CNN accelerators fully utilize the normally-off characteristics of memristor crossbars to achieve instant network precision tuning without worrying about the model reloading penalty. The ReRAM-based accelerator, with the proposed neural parameter mapping policy and the novel mixed-model training method, induces negligible precision-switching latency and power consumption when compared with traditional variable precision accelerators. In evaluation with state-of-the-art workloads, the proposed ReRAM DNN architecture saves 58.3%-62.47% area overhead over the baseline design. We also leverage the proposed ReRAM accelerator architecture to build a novel always-on Key-Word Spotting (KWS) system. The KWS design can switch between different precision modes to capture the relevant sound with high accuracy. The experimental results show the precisionadjustable KWS architecture saves considerable operating energy when fed with realistic test-sets of audio data.

Two-stage streaming keyword detection and localization with multi-scale depthwise temporal convolution

A keyword spotting (KWS) system running on smart devices should accurately detect the appearances and predict the locations of predefined keywords from audio streams, with small footprint and high efficiency. To this end, this paper proposes a new two-stage KWS method which combines a novel multi-scale depthwise temporal convolution (MDTC) feature extractor and a two-stage keyword detection and localization module. The MDTC feature extractor learns multi-scale feature representation efficiently with dilated depthwise temporal convolution, modeling both the temporal context and the speech rate variation. We use a region proposal network (RPN) as the first-stage KWS. At each frame, we design multiple time regions, which all take the current frame as the end position but have different start positions. These time regions (or formally anchors) are used to indicate rough location candidates of keyword. With frame level features from the MDTC feature extractor as inputs, RPN learns to propose keyword region proposals based on the designed anchors. To alleviate the keyword/non-keyword class imbalance problem, we specifically introduce a hard example mining algorithm to select effective negative anchors in RPN training. The keyword region proposals from the first-stage RPN contain keyword location information which is subsequently used to explicitly extract keyword related sequential features to train the second-stage KWS. The second-stage system learns to classify and transform region proposal to keyword IDs and ground-truth keyword region respectively. Experiments on the Google Speech Command dataset show that the proposed MDTC feature extractor surpasses several competitive feature extractors with a new state-of-the-art command classification error rate of 1.74%. With the MDTC feature extractor, we further conduct wake-up word (WuW) detection and localization experiments on a commercial WuW dataset. Compared to a strong baseline, our proposed two-stage method achieves relatively 27–32% better false rejection rate at one false alarm per hour, while for keyword localization, the two-stage approach achieves more than 0.95 mean intersection-over-union ratio, which is clearly better than the one-stage RPN method.

Ultralow Power Feature Extractor Using Switched-Capacitor-Based Bandpass Filter, Max Operator, and Neural Network Processor for Keyword Spotting

This letter presents an ultralow power keyword spotting (KWS) system using a feature extractor comprising a bandpass filter, a max operator, and a time-delay neural network (TDNN) processor based on a switched capacitor technique. In the proposed KWS system, TDNN is used as a classifier. The first layer of the classifier requires an accurate calculation, whereas the rest of the layers can be binarized. Thus, in this study, feature extraction and the first layer of TDNN are performed in the analog domain, while the remainder is processed in the digital domain. The proposed architecture can remove high-precision ADC, which achieves ultralow power KWS. The proposed feature extractor is fabricated in the 65-nm CMOS process. The measurement results show that the proposed KWS system consumes 270 nW to detect two keywords.