Wake-up Word Research Articles

Low Power but High Energy: The Looming Costs of Billions of Smart Devices

Individually, wall-powered Internet of Things devices are small: in form factor, in complexity, in function, and in power draw. However, at scale, and certainly at the scale optimistic forecasters project, these small devices add up to be a big energy problem. Just adding a single two watt sensor to each US building would add to more annual energy consumption than some small countries. Wall-powered IoT devices are also easier to create than their energy-constrained (i.e. battery-powered) counterparts, and marketed as more convenient (no hub required!), leading to their continued growth. Yet, unlike other energy consuming devices, there are no Energy Star (or equivalent) standards for smart devices. Despite having very infrequent active times, they draw power for functions like AC-DC conversion, wireless communication, and wakeup word detection continuously. Further, the discrete nature of devices and siloed nature of IoT ecosystems leads to significant redundancy in IoT devices. We posit that new techniques are needed to reverse this trend. This includes new techniques for auditing devices, systems that leverage existing devices rather than requiring new ones, and architectures that have less reliance on the cloud (and the energy overhead of network usage and cloud compute). The IoT is pitched to improve energy efficiency and reduce users' carbon footprints, but we need a new research agenda to ensure the devices themselves are not the next problem.

FPGA Implementation of Keyword Spotting System Using Depthwise Separable Binarized and Ternarized Neural Networks.

Keyword spotting (KWS) systems are used for human-machine communications in various applications. In many cases, KWS involves a combination of wake-up-word (WUW) recognition for device activation and voice command classification tasks. These tasks present a challenge for embedded systems due to the complexity of deep learning algorithms and the need for optimized networks for each application. In this paper, we propose a depthwise separable binarized/ternarized neural network (DS-BTNN) hardware accelerator capable of performing both WUW recognition and command classification on a single device. The design achieves significant area efficiency by redundantly utilizing bitwise operators in the computation of the binarized neural network (BNN) and ternary neural network (TNN). In a complementary metal-oxide semiconductor (CMOS) 40 nm process environment, the DS-BTNN accelerator demonstrated significant efficiency. Compared with a design approach where BNN and TNN were independently developed and subsequently integrated as two separate modules into the system, our method achieved a 49.3% area reduction while yielding an area of 0.558 mm2. The designed KWS system, which was implemented on a Xilinx UltraScale+ ZCU104 field-programmable gate array (FPGA) board, receives real-time data from the microphone, preprocesses them into a mel spectrogram, and uses this as input to the classifier. Depending on the order, the network operates as a BNN or a TNN for WUW recognition and command classification, respectively. Operating at 170 MHz, our system achieved 97.1% accuracy in BNN-based WUW recognition and 90.5% in TNN-based command classification.

Multichannel Speech Enhancement of Target Speaker Based on Wakeup Word Mask Estimation with Deep Neural Network

Multichannel Speech Enhancement of Target Speaker Based on Wakeup Word Mask Estimation with Deep Neural Network

Speech Assistant System With Local Client and Server Devices to Guarantee Data Privacy

Users of speech assistant systems have reservations about the distributed approach of these systems. They have concerns that people might get access to the transmitted speech data or that somebody is able to access their microphone from outside. Therefore, we investigate the concept of a setup with local client and server systems. This comes along with the requirement of cost-efficient realizations of client and server. We examined a number of different cost-efficient server solutions depending on the required recognition capability of specific applications. A fairly cost-efficient solution is the use of a small computing device for recognizing a few dozens of words with a GMM-HMM based recognition. To perform a DNN-HMM based recognition, we looked at small computing devices with an integrated additional graphical processor unit (GPU). Furthermore, we investigated the use of low-cost PCs for implementing real-time versions of the Kaldi framework to allow the recognition of large vocabularies. We investigated the control of a smart home by speech as an exemplary application. For this, we designed compact client systems that can be integrated at certain places inside a room, e.g., in a standard outlet socket. Besides activating a client by a sensor that detects approaching people, the recognition of a spoken wake-up word is the usual way for activation. We developed a keyword recognition algorithm that can be implemented in the client despite its limited computing resources. The control of the whole dialogue has been integrated in our client, so that no further server is needed. In a separate study, we examined the approach of an extremely energy-efficient realization of the client system without the need of an external power supply. The approach is based on using a special microphone with an additional low-power operating mode detecting the exceeding of a preset sound level threshold only. This detection can be used to wake up the client's microcontroller and to make the microphone switch to normal operating mode. In the listening mode, the energy consumption of the microphone is so low that a client system can be active for months with an energy supply from standard batteries only.

Two-stage streaming keyword detection and localization with multi-scale depthwise temporal convolution

A keyword spotting (KWS) system running on smart devices should accurately detect the appearances and predict the locations of predefined keywords from audio streams, with small footprint and high efficiency. To this end, this paper proposes a new two-stage KWS method which combines a novel multi-scale depthwise temporal convolution (MDTC) feature extractor and a two-stage keyword detection and localization module. The MDTC feature extractor learns multi-scale feature representation efficiently with dilated depthwise temporal convolution, modeling both the temporal context and the speech rate variation. We use a region proposal network (RPN) as the first-stage KWS. At each frame, we design multiple time regions, which all take the current frame as the end position but have different start positions. These time regions (or formally anchors) are used to indicate rough location candidates of keyword. With frame level features from the MDTC feature extractor as inputs, RPN learns to propose keyword region proposals based on the designed anchors. To alleviate the keyword/non-keyword class imbalance problem, we specifically introduce a hard example mining algorithm to select effective negative anchors in RPN training. The keyword region proposals from the first-stage RPN contain keyword location information which is subsequently used to explicitly extract keyword related sequential features to train the second-stage KWS. The second-stage system learns to classify and transform region proposal to keyword IDs and ground-truth keyword region respectively. Experiments on the Google Speech Command dataset show that the proposed MDTC feature extractor surpasses several competitive feature extractors with a new state-of-the-art command classification error rate of 1.74%. With the MDTC feature extractor, we further conduct wake-up word (WuW) detection and localization experiments on a commercial WuW dataset. Compared to a strong baseline, our proposed two-stage method achieves relatively 27–32% better false rejection rate at one false alarm per hour, while for keyword localization, the two-stage approach achieves more than 0.95 mean intersection-over-union ratio, which is clearly better than the one-stage RPN method.

TASE: Task-Aware Speech Enhancement for Wake-Up Word Detection in Voice Assistants

Wake-up word spotting in noisy environments is a critical task for an excellent user experience with voice assistants. Unwanted activation of the device is often due to the presence of noises coming from background conversations, TVs, or other domestic appliances. In this work, we propose the use of a speech enhancement convolutional autoencoder, coupled with on-device keyword spotting, aimed at improving the trigger word detection in noisy environments. The end-to-end system learns by optimizing a linear combination of losses: a reconstruction-based loss, both at the log-mel spectrogram and at the waveform level, as well as a specific task loss that accounts for the cross-entropy error reported along the keyword spotting detection. We experiment with several neural network classifiers and report that deeply coupling the speech enhancement together with a wake-up word detector, e.g., by jointly training them, significantly improves the performance in the noisiest conditions. Additionally, we introduce a new publicly available speech database recorded for the Telefónica’s voice assistant, Aura. The OK Aura Wake-up Word Dataset incorporates rich metadata, such as speaker demographics or room conditions, and comprises hard negative examples that were studiously selected to present different levels of phonetic similarity with respect to the trigger words “OK Aura”.

Triplet loss based domain adversarial training for robust wake-up word detection in noisy environments

Triplet loss based domain adversarial training for robust wake-up word detection in noisy environments

Self-Defined Text-Dependent Wake-Up-Words Speaker Recognition System

In recent years, wake-up-words (WUW) technology is highly developed in some speaker recognition system. It is the progress of verifying a person’s claimed identity from their voice characteristics, and can be efficiently deployed in some consumer applications. In this paper, we proposed a self-defined text-dependent wake-up-words (WUW) speaker recognition system and its implementation. The whole system is divided into two phases: training phase and testing phase. In the training phase, a wake-up word by language is recorded, and the voice segment is cut out by using Voice Activity Detection (VAD). Then we use the Mel-Frequency Cepstral Coefficients (MFCC) as the pre-processing to extract the speech features. After obtaining the speech features, we use Gaussian Mixture Model (GMM) and Hidden Markov Model (HMM) simultaneously for training. In the testing phase, we build GMM and HMM continuously and use the Levenshtein Distance (LD) to calculate the differences of the state sequences between the dataset and the unknown speech input. If the unknown speech input passes the threshold, then it means a wake-up event is derived. The experimental results show that the average accuracy is 93.31 %, 82.42% and 3.38 % in 10dB, 5dB and 0dB of Signal Noise Ratio (SNR) respectively. The CPU and memory usage of entire system is around 757 MIPS and 40MB respectively.

On-Device System for Device Directed Speech Detection for Improving Human Computer Interaction

Voice assistants (VA) are finding a place in many households, with increasing numbers. Nevertheless, for every interaction user invokes VA using a key or wake-up word, which is too common and hinders natural conversation. To solve this, we propose an On-Device solution that listens to the user continuously for only predefined period and classifies the utterance into device-directed or non-device-directed using a deep learning-based model. Since our solution is On-Device, the privacy of the user is maintained. We tried to solve false acceptance of utterance as a command and incorrect rejection of the command as background noise to improve user experience. We calculated the False Acceptance Rate (FAR) and False Rejection Rate (FRR) for different thresholds to evaluate our model. Then we calculated Equal Error Rate (EER) results with the FRR and FAR data. We achieved 3.6% of EER on our best optimized Stage-1 model and EER of 5.1% on Stage-2 model. We have also used accuracy as evaluation metrics during training and testing. We achieved an accuracy of 96.8% on testing data with our Stage-1 model and 95.3% on Stage-2 model.

Small-Footprint Wake Up Word Recognition in Noisy Environments Employing Competing-Words-Based Feature

This paper proposes a small-footprint wake-up-word (WUW) recognition system for real noisy environments by employing the competing-words-based feature. Competing-words-based features are generated using a ResNet-based deep neural network with small parameters using the competing-words dataset. The competing-words dataset consists of the most acoustically similar and dissimilar words to the WUW used for our system. The obtained features are used as input to the classification network, which is developed using the convolutional neural network (CNN) model. To obtain sufficient data for training, data augmentation is performed by using a room impulse response filter and adding sound signals of various television shows as background noise, which simulates an actual living room environment. The experimental results demonstrate that the proposed WUW recognition system outperforms the baselines that employ CNN and ResNet models. The proposed system shows 1.31% in equal error rate and 1.40% false rejection rate at a 1.0% false alarm rate, which are 29.57% and 50.00% relative improvements compared to the ResNet system, respectively. The number of parameters used for the proposed system is reduced by 83.53% compared to the ResNet system. These results prove that the proposed system with the competing-words-based feature is highly effective at improving WUW recognition performance in noisy environments with a smaller footprint.

Voice interaction on TV: analysis of natural language interaction models and recommendations for voice user interfaces

The goal of this study was to perform an evaluation of a set of voice interaction models (supported by a hands-free solution activated by a wake-up word, a mobile app and a TV remote control with microphone) to identify the most appropriate solution for interactive television. The research addressed issues associated with natural language systems such as usability, interaction and privacy perception, and aimed to analyze the strengths and limitations of the voice interaction models. On a first evaluation approach, a prototype based on a Wizard-of-Oz methodology was used, while a second approach was based on a functional prototype. The preferred interaction model was the hands-free solution activated by a wake-up word because it was easy to use and raised the least difficulties in any task execution. Despite this result, the other two models are not disregarded for a future voice interaction system in television. The TV remote control was the most natural way of interaction for the study’s participants. The need for control provided by the remote and by the app makes the participants feel like these grant more privacy. Participants considered that a voice-operated system for TV would be very useful and almost all were receptive to having such a system at home. Lastly, based on commercial standards and guidelines, solutions to issues identified by participants in the visual interface of the TV system were proposed and considered for the next phase of prototype development, also benefiting other researches in the field.

Cross-Informed Domain Adversarial Training for Noise-Robust Wake-Up Word Detection

A proper representation that can well express the characteristics of a word plays an important role in wake-up word detection (WWD). However, it may be easily corrupted due to various types of environmental noise occurred in the place where WWD typically works, causing unreliable performance. To deal with this practical issue, we propose a novel strategy called cross-informed domain adversarial training (CiDAT) for noise-robust WWD. In the method, additional paths were introduced to conventional domain adversarial training (DAT) to encourage its ability to generate domain-invariant representation. Experiments on the Aurora4 corpus verified that CiDAT significantly outperformed the baselines as well as conventional DAT.

Interlayer Selective Attention Network for Robust Personalized Wake-Up Word Detection

Previous research methods on wake-up word detection (WWD) have been proposed with focus on finding a decent word representation that can well express the characteristics of a word. However, there are various obstacles such as noise and reverberation which make it difficult in real-world environments where WWD works. To tackle this, we propose a novel architecture called interlayer selective attention network (ISAN) which generates more robust word representation by introducing the concept of selective attention. Experiments in real-world scenarios demonstrated that the proposed ISAN outperformed several baseline methods as well as other attention methods. In addition, the effectiveness of ISAN was analyzed with visualizations.

Speech Processing for Digital Home Assistants: Combining Signal Processing With Deep-Learning Techniques

Once a popular theme of futuristic science fiction or far-fetched technology forecasts, digital home assistants with a spoken language interface have become a ubiquitous commodity today. This success has been made possible by major advancements in signal processing and machine learning for so-called far-field speech recognition, where the commands are spoken at a distance from the sound-capturing device. The challenges encountered are quite unique and different from many other use cases of automatic speech recognition (ASR). The purpose of this article is to describe, in a way that is amenable to the nonspecialist, the key speech processing algorithms that enable reliable, fully hands-free speech interaction with digital home assistants. These technologies include multichannel acoustic echo cancellation (MAEC), microphone array processing and dereverberation techniques for signal enhancement, reliable wake-up word and end-of-interaction detection, and high-quality speech synthesis as well as sophisticated statistical models for speech and language, learned from large amounts of heterogeneous training data. In all of these fields, deep learning (DL) has played a critical role.

Performance of WUW and general ASR speech recognition systems in different acoustic environments

The Automatic Speech Recognition (ARS) Systems have become popular in the different areas of applications, such as education assistance, medical assistance, personal assistant, robotics and vehicles, telecommunications Industry, disabilities systems, home automation, and security access control. All these applications in life will increase the number of speech commands, which are Wake-up-words. The WUW speech recognition is similar to Key-Word spotting. However, in this paper, we propose an approach that will be used to detect the Wake-up-word (WUW) as a command with 100% accuracy and to provide a whole speech recognition system that can work in both WUW and General ASR systems with high accuracy. Also, this new ASR system will be used to solve one of the biggest problems in speech recognition, which is how to discriminate between the uses of a word or phrase in an alerting and a referential context. For example, in the use of the word “Computer'' in an alerting context is Computer, show me the chart? '' and in a referential context it is “Every computer should have a speaker. Moreover, we tested the new system with different acoustic environments, such as different background noise levels, different speaker distances to the microphone, and different speakers.

Compact Wake-Up Word Speech Recognition on Embedded Platforms

The wake-up word speech recognition system is a new paradigm in the field of automatic speech recognition (ASR). This new paradigm is not yet widely recognized but useful in many applications such as mobile phones and smart home systems. In this paper we describe the development of a compact wake-up word recognizer for embedded platforms. To keep resource cost low, a variety of simplification techniques are used. Speech feature observations are compressed to lower dimension and the simple distance-based template matching method is used in place of complex Viterbi scoring. We apply double scoring method to achieve a better performance. To cooperate with double scoring method, the support vector machine classifier is used as well. We were able to accomplish a performance improvement with false rejection rate reduced from 6.88% to 5.50% and false acceptance rate reduced from 8.40% to 3.01%.

A novel Wake-Up-Word speech recognition system, Wake-Up-Word recognition task, technology and evaluation

Wake-Up-Word (WUW) is a new paradigm in speech recognition (SR) that is not yet widely recognized. This paper defines and investigates WUW speech recognition, describes details of this novel solution and the technology that implements it. WUW SR is defined as detection of a single word or phrase when spoken in the alerting context of requesting attention, while rejecting all other words, phrases, sounds, noises and other acoustic events and the same word or phrase spoken in non-alerting context with virtually 100% accuracy. In order to achieve this accuracy, the following innovations were accomplished: (1) Hidden Markov Model triple scoring with Support Vector Machine classification, (2) Combining multiple speech feature streams: Mel-scale Filtered Cepstral Coefficients (MFCCs), Linear Prediction Coefficients (LPC)-smoothed MFCCs, and Enhanced MFCC, and (3) Improved Voice Activity Detector with Support Vector Machines. WUW detection and recognition performance is 2514%, or 26 times better than HTK for the same training & testing data, and 2271%, or 24 times better than Microsoft SAPI 5.1 recognizer. The out-of-vocabulary rejection performance is over 65,233%, or 653 times better than HTK, and 5900% to 42,900%, or 60 to 430 times better than the Microsoft SAPI 5.1 recognizer. This solution that utilizes a new recognition paradigm applies not only to WUW task but also to any general Speech Recognition tasks.