Keyword Spotting Research Articles

Wav2KWS: Transfer Learning From Speech Representations for Keyword Spotting

With the expanding development of on-device artificial intelligence, voice-enabled devices such as smart speakers, wearables, and other on-device or edge processing systems have been proposed. However, building or obtaining large training datasets that are essential for robust keyword spotting (KWS) remains cumbersome. To address this problem, we propose a deep neural network that can rapidly establish a high-performance KWS system from arbitrary keyword instruction sets. We use an encoder pretrained with a large-scale speech corpus as the backbone network and then design an effective transfer network for KWS. To demonstrate the feasibility of the proposed network, various experiments were conducted on Google Speech Command Datasets V1 and V2. In addition, to verify the applicability of the network for different languages, we conducted experiments using three different Korean speech command datasets. The proposed network outperforms state-of-the-art deep neural networks in both experiments. Furthermore, the proposed network can understand real human voice even when trained with synthetic text-to-speech data.

TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech

We introduce a self-supervised speech pre-training method called TERA, which stands for Transformer Encoder Representations from Alteration. Recent approaches often learn by using a single auxiliary task like contrastive prediction, autoregressive prediction, or masked reconstruction. Unlike previous methods, we use alteration along three orthogonal axes to pre-train Transformer Encoders on a large amount of unlabeled speech. The model learns through the reconstruction of acoustic frames from their altered counterpart, where we use a stochastic policy to alter along various dimensions: time, frequency, and magnitude. TERA can be used for speech representations extraction or fine-tuning with downstream models. We evaluate TERA on several downstream tasks, including phoneme classification, keyword spotting, speaker recognition, and speech recognition. We present a large-scale comparison of various self-supervised models. TERA achieves strong performance in the comparison by improving upon surface features and outperforming previous models. In our experiments, we study the effect of applying different alteration techniques, pre-training on more data, and pre-training on various features. We analyze different model sizes and find that smaller models are strong representation learners than larger models, while larger models are more effective for downstream fine-tuning than smaller models. Furthermore, we show the proposed method is transferable to downstream datasets not used in pre-training.

Претраживе дигиталне рукописне колекције: могућност за рашчитавање српске ћирилице

Последњих неколико година, библиотеке и архиви, схватајући значај дигитализације, првенствено у погледу доступности садржаја, све више своју богату рукописну грађу преводе у дигитални формат како би, са једне стране, постала доступна корисницима, а са друге стране, како би је сачували од пропадања. Концепт виртуелног истраживачког окружења, настао је као део Пројекта за препознавање и обогаћивање архивских докумената (Recognition and Enrichment of Archival Documents – H2020 READ) и има потенцијал да омогући потпуно нови приступ историјским рукописним документима који се чувају у институцијама културе широм Европе. Главни циљ READ пројекта био је да се изгради виртуелно истраживачко окружење у оквиру кога би се развијале врхунске технологије за аутоматско препознавање, транскрипцију, индексирање и обогаћивање рукописних архивских докумената. Универзитетска библиотека „Светозар Марковић“ се на самом почетку укључила у овај пројекат, као придружени партнер, са циљем да се развије алат који ће омогућити рашчитавање српске ћириличне рукописне грађе.

Unsupervised Pre-Training for Voice Activation

The problem of voice activation is to find a pre-defined word in the audio stream. Solutions such as keyword spotter “Ok, Google” for Android devices or keyword spotter “Alexa” for Amazon devices use tens of thousands to millions of keyword examples in training. In this paper, we explore the possibility of using pre-trained audio features to build voice activation with a small number of keyword examples. The contribution of this article consists of two parts. First, we investigate the dependence of the quality of the voice activation system on the number of examples in training for English and Russian and show that the use of pre-trained audio features, such as wav2vec, increases the accuracy of the system by up to 10% if only seven examples are available for each keyword during training. At the same time, the benefits of such features become less and disappear as the dataset size increases. Secondly, we prepare and provide for general use a dataset for training and testing voice activation for the Lithuanian language. We also provide training results on this dataset.

A Model for Evaluating the Performance of a Multiple Keywords Spotting System for the Transcription of Historical Handwritten Documents.

This paper proposes a performance model for estimating the user time needed to transcribe small collections of handwritten documents using a keyword spotting system (KWS) that provides a number of possible transcriptions for each word image. The model assumes that only information obtained from a small training set is available, and establishes the constraints on the performance measures to achieve a reduction of the time for transcribing the content with respect to the time required by human experts. The model is complemented with a procedure for computing the parameters of the model and eventually estimating the improvement of the time to achieve a complete and error-free transcription of the documents.

A 0.44-μJ/dec, 39.9-μs/dec, Recurrent Attention In-Memory Processor for Keyword Spotting

This article presents a deep learning-based classifier IC for keyword spotting (KWS) in 65-nm CMOS designed using an algorithm-hardware co-design approach. First, a recurrent attention model (RAM) algorithm for the KWS task (the KeyRAM algorithm) is proposed. The KeyRAM algorithm enables accuracy versus energy scalability via a confidence-based computation (CC) scheme, leading to a 2.5× reduction in computational complexity compared to state-of-the-art (SOTA) neural networks, and is well-suited for in-memory computing (IMC) since the bulk (89%) of its computations are 4-b matrix-vector multiplies. The KeyRAM IC comprises a multi-bit multi-bank IMC architecture with a digital co-processor. A sparsity-aware summation scheme is proposed to alleviate the challenge faced by IMCs when summing sparse activations. The digital co-processor employs diagonal major weight storage to compute without any stalls. This combination of the IMC and digital processors enables a balanced tradeoff between energy efficiency and high accuracy computation. The resultant KWS IC achieves SOTA decision latency of 39.9 μs with a decision energy <; 0.5 μJ/dec which translates to more than 24 × savings in the energy-delay product (EDP) of decisions over existing KWS ICs.

A Pitch and Noise Robust Keyword Spotting System Using SMAC Features with Prosody Modification

Spotting of keywords in continuous speech signal with the aid of the computer is called a keyword spotting (KWS) system. A variety of strategies have been suggested in the literature to detect keywords from the adult’s speech effectively. However, only a limited number of studies have been reported for KWS in children’s speech. Due to the difference in physiological properties, the pitch and speaking rate of children’s differ from the adult’s. Consequently, KWS system model parameters trained on the speech data from adult’s signal yield poor performance for children speech. In this paper, we have developed a KWS system for spotting keywords from children’s speech using models trained on adults’ speech. The proposed approach uses spectral moment time–frequency distribution augmented by low-order cepstral (SMAC) as the front-end feature. The mismatches due to differences in pitch and speaking rate of children and adult speakers are further mitigated by data-augmented training using explicit pitch and speaking rate modifications. The experimental findings presented in this paper show that the SMAC feature offers significantly better output for both clean and noisy test conditions than the conventional Mel frequency cepstral coefficients.

A 510-nW Wake-Up Keyword-Spotting Chip Using Serial-FFT-Based MFCC and Binarized Depthwise Separable CNN in 28-nm CMOS

We propose a sub-μW always-ON keyword spotting (μKWS) chip for audio wake-up systems. It is mainly composed of a neural network (NN) and a feature extraction (FE) circuit. For significantly reducing the memory footprint and computational load, four techniques are used to achieve ultralow-power consumption: 1) a serial-FFT-based Mel-frequency cepstrum coefficient circuit is designed for FE, instead of the common parallel FFT. 2) A small-sized binarized depthwise separable convolutional NN (DSCNN) is designed as the classifier. 3) A framewise incremental computation technique is devised in contrast to the conventional whole-word processing. 4) Reduced computation allows a low system clock frequency, which enables near-threshold voltage operation, and low leakage memory blocks are designed to minimize the leakage power. Implemented in 28-nm CMOS technology, this μKWS consumes 0.51 μW at a 40-kHz frequency and a 0.41-V supply, with an area of 0.23 mm2. Using the Google speech command data set, 97.3% accuracy is reached for a one-word KWS task and 94.6% for a two-word task.

One Step Is Not Enough: A Multi-Step Procedure for Building the Training Set of a Query by String Keyword Spotting System to Assist the Transcription of Historical Document

Digital libraries offer access to a large number of handwritten historical documents. These documents are available as raw images and therefore their content is not searchable. A fully manual transcription is time-consuming and expensive while a fully automatic transcription is cheaper but not comparable in terms of accuracy. The performance of automatic transcription systems is strictly related to the composition of the training set. We propose a multi-step procedure that exploits a Keyword Spotting system and human validation for building up a training set in a time shorter than the one required by a fully manual procedure. The multi-step procedure was tested on a data set made up of 50 pages extracted from the Bentham collection. The palaeographer that transcribed the data set with the multi-step procedure instead of the fully manual procedure had a time gain of 52.54%. Moreover, a small size training set that allowed the keyword spotting system to show a precision value greater than the recall value was built with the multi-step procedure in a time equal to 35.25% of the time required for annotating the whole data set.

UltraTrail: A Configurable Ultralow-Power TC-ResNet AI Accelerator for Efficient Keyword Spotting

Recent advances in machine learning show the superior behavior of temporal convolutional networks (TCNs) and especially their combination with residual networks (TC-ResNet) for intelligent sensor signal processing in comparison to classical CNNs and LSTMs. In this article, we propose UltraTrail, a configurable, ultralow-power TC-ResNet AI accelerator for sensor signal processing and its application to efficient keyword spotting (KWS). Following a strict hardware/model co-design approach, we have derived an optimized low-power hardware architecture for generalized TC-ResNet topologies consisting of a configurable array of processing elements and a distributed memory with dynamic content reallocation. We additionally extend the network with conditional computing to reduce the number of operations during inference and to provide the possibility for power-gating. The final accelerator implementation in Globalfoundries' 22FDX technology achieves a power consumption of 8.2 μW for the task of always-on KWS meeting the real-time requirement of 100 ms per inference with an accuracy of 93% on the Google Speech Command Dataset.

Automatic CNN-Based Arabic Numeral Spotting and Handwritten Digit Recognition by Using Deep Transfer Learning in Ottoman Population Registers

Historical manuscripts and archival documentation are handwritten texts which are the backbone sources for historical inquiry. Recent developments in the digital humanities field and the need for extracting information from the historical documents have fastened the digitization processes. Cutting edge machine learning methods are applied to extract meaning from these documents. Page segmentation (layout analysis), keyword, number and symbol spotting, handwritten text recognition algorithms are tested on historical documents. For most of the languages, these techniques are widely studied and high performance techniques are developed. However, the properties of Arabic scripts (i.e., diacritics, varying script styles, diacritics, and ligatures) create additional problems for these algorithms and, therefore, the number of research is limited. In this research, we first automatically spotted the Arabic numerals from the very first series of population registers of the Ottoman Empire conducted in the mid-nineteenth century and recognized these numbers. They are important because they held information about the number of households, registered individuals and ages of individuals. We applied a red color filter to separate numerals from the document by taking advantage of the structure of the studied registers (numerals are written in red). We first used a CNN-based segmentation method for spotting these numerals. In the second part, we annotated a local Arabic handwritten digit dataset from the spotted numerals by selecting uni-digit ones and tested the Deep Transfer Learning method from large open Arabic handwritten digit datasets for digit recognition. We achieved promising results for recognizing digits in these historical documents.

Bonseyes AI Pipeline—Bringing AI to You

Next generation of embedded Information and Communication Technology (ICT) systems are interconnected and collaborative systems able to perform autonomous tasks. The remarkable expansion of the embedded ICT market, together with the rise and breakthroughs of Artificial Intelligence (AI), have put the focus on the Edge as it stands as one of the keys for the next technological revolution: the seamless integration of AI in our daily life. However, training and deployment of custom AI solutions on embedded devices require a fine-grained integration of data, algorithms, and tools to achieve high accuracy and overcome functional and non-functional requirements. Such integration requires a high level of expertise that becomes a real bottleneck for small and medium enterprises wanting to deploy AI solutions on the Edge , which, ultimately, slows down the adoption of AI on applications in our daily life. In this work, we present a modular AI pipeline as an integrating framework to bring data, algorithms, and deployment tools together. By removing the integration barriers and lowering the required expertise, we can interconnect the different stages of particular tools and provide a modular end-to-end development of AI products for embedded devices. Our AI pipeline consists of four modular main steps: (i) data ingestion, (ii) model training, (iii) deployment optimization, and (iv) the IoT hub integration. To show the effectiveness of our pipeline, we provide examples of different AI applications during each of the steps. Besides, we integrate our deployment framework, Low-Power Deep Neural Network (LPDNN), into the AI pipeline and present its lightweight architecture and deployment capabilities for embedded devices. Finally, we demonstrate the results of the AI pipeline by showing the deployment of several AI applications such as keyword spotting, image classification, and object detection on a set of well-known embedded platforms, where LPDNN consistently outperforms all other popular deployment frameworks.

Portraying Screen Reading Behavior of College Students of University of Delhi

The present paper explored various screen reading activities of undergraduate students of twelve colleges of University of Delhi. It included browsing/scanning, keyword spotting, concentrated reading, one time reading, reading selectively and non-linear reading. It also investigated several ways of handling the electronic documents along with frequently used reading devices and preferred location for reading the documents. The study administered structured print questionnaires and received 506 completed questionnaires from undergraduate students of different disciplines i.e. sciences, social sciences and humanities. Cronbach’s Alpha Reliability Coefficient test was applied using the SPSS (Statistical Package for Social Science, Version-25) along with the Weighted Mean. Results revealed that Screen reading activities such as browsing/scanning (54.0 %), reading selectively (51.0 %), concentrated reading (36.2 %), one time reading (33.0 %), keyword spotting (28.3 %) and non-linear reading (23.5 %) were increasing among undergraduate students of all the three disciplines. The preferences of undergraduate students for reading devices and locations reveals that majority of 51.0 per cent students used smartphones followed by laptops (29.5 %), personal computers (7.4 %) and the minimum number of only 5.5 per cent of students were using tablets. Maximum number of 38.3 per cent students preferred reading at their homes followed by library (17.8 %) and while travelling (17.2 %). The lowest number of (6.8 %) students read books in their hostels. Overall, it revealed that in present scenario, students’ concentration level has found increased reading the content on screen. Whereas, the situation was other way round with the earlier generations. Few factors such as convenience, flexibility, portability and ownership in using smart devices make them connected online with the peer groups and read content on screen at their own pace.

Zone-based keyword spotting in Bangla and Devanagari documents

In this paper, we present a word spotting system in text lines for offline Indic scripts such as Bangla (Bengali) and Devanagari. Recently, it was shown that the zone-wise recognition method improves word recognition performance than the conventional full word recognition system in Indic scripts, like Bangla, Devanagari, Gurumukhi (Roy et al. in Pattern Recogn 60: 1057-1075, 26; Bhunia et al. in Pattern Recogn 79: 12–31, 6). Inspired from this idea we consider the zone segmentation approach and use middle zone information to improve the traditional word spotting performance. To avoid the problem of zone segmentation using heuristic approach, we propose here a new HMM based approach to segment the upper and lower zone components from the text line images. The candidate keywords are searched from a line without segmenting characters or words. Also, we propose a feature combining foreground and background information of text line images for keyword-spotting by character filler models. A significant improvement in performance is noted by using both foreground and background information instead of the individual one. Pyramid Histogram of Oriented Gradient (PHOG) feature has been used in our word spotting framework. From the experiment, it has been noted that the proposed zone-segmentation based system outperforms traditional approaches of word spotting.

Improving approximate neural networks for perception tasks through specialized optimization

Approximate Computing has been proven successful in reducing the energy consumption of Deep Neural Networks (DNNs) implemented in embedded systems. For efficient DNN approximation at software and hardware levels, a specialized simulation environment and optimization methodology are required, to reduce execution and optimization times, as well as to maximize energy savings. Traditional frameworks for cross-layer approximate computation of DNNs are generally built only for simulation of convolutional and fully-connected layers, limiting the DNN types to be optimized through approximations. In this work, we present a specialized simulation environment for approximate DNNs, which allows for optimization of several DNN architectures built with more complex DNN layers such as depthwise convolutions and Recurrent Neural Units (RNNs) for time series processing. Low execution time overhead is achieved hereby through efficient GPU acceleration. Additionally, we deliver an analysis of approximate DNN and RNN robustness against quantization noise and different approximation levels. Finally, through specialized approximate retraining, we achieve promising energy savings and negligible accuracy losses with highly complex DNNs for image classification with ImageNet, such as MobileNet, and RNNs for keyword spotting with the Speech Commands Dataset.

A depthwise separable convolutional neural network for keyword spotting on an embedded system

A keyword spotting algorithm implemented on an embedded system using a depthwise separable convolutional neural network classifier is reported. The proposed system was derived from a high-complexity system with the goal to reduce complexity and to increase efficiency. In order to meet the requirements set by hardware resource constraints, a limited hyper-parameter grid search was performed, which showed that network complexity could be drastically reduced with little effect on classification accuracy. It was furthermore found that quantization of pre-trained networks using mixed and dynamic fixed point principles could reduce the memory footprint and computational requirements without lowering classification accuracy. Data augmentation techniques were used to increase network robustness in unseen acoustic conditions by mixing training data with realistic noise recordings. Finally, the system’s ability to detect keywords in a continuous audio stream was successfully demonstrated.

AN EVALUATION OF SOME FACTORS AFFECTING ACCURACY OF THE VIETNAMESE KEYWORD SPOTTING SYSTEM

Keyword spotting (KWS) is one of the important systems on speech applications, such as data mining, call routing, call center, customer-controlled smartphone, smart home systems with voice control, etc. With the goals of researching some factors affecting the Vietnamese Keyword spotting system, we study the combination architecture of CNN (Convolutional Neural Networks)-RNN (Recurrent Neural Networks) on both clean and noise environments with 2 distance speaker cases: 1m and 2m. The obtained results show that the noise trained models are better performance than clean trained models in any (clean or noise) testing environment. The results in this far-field experiment suggest to us how to choose the suitable distance of the recording microphones to the speaker so that there is no redundancy of data with the contexts considered to be the same.

A 22nm, 10.8 μ W/15.1 μ W Dual Computing Modes High Power-Performance-Area Efficiency Domained Background Noise Aware Keyword- Spotting Processor

This paper proposes a high power-performance-area efficient background noise aware keyword-spotting (KWS) processor based on an optimized binarized weight network (BWN). To reduce the power consumption while maintaining the system recognition accuracy for different background noise, the KWS processor with a SNR prediction module can be adaptively configured to use dual computing modes (standard computing mode and approximate computing mode) for both high recognition accuracy under high background noise and ultra-low power consumption under low background noise. The mel-scale frequency cepstral coefficients (MFCC) module is optimized with approximate computing technologies, which can reduce the power consumption by up to $3.1\times $ and $5.7\times $ for high/low background noise, respectively. Based on the evaluation of the architecture design space exploration, an ultra-low power BWN accelerator with low voltage, area and leakage power and using precision self-adaptive approximate computing units was proposed. Evaluated under 22nm process technology, this work can support up to 10 keywords real time recognition with power consumption of $15.1~\mu \text{W}$ for high background noise and $10.8~\mu \text{W}$ for low background noise. Compared to the state-of-the-art KWS architectures, our work can achieve ultra-low power consumption (about $1.7\times $ reduced), while maintaining high system capability and adaptability.

An End-to-End Model Based on Multiple Neural Networks with Data Augmentation for Keyword Spotting

In this paper, we propose a network for small footprint keyword spotting. It includes four parts, data augmentation, Time-Delay Neural Network (TDNN) and Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and attention mechanism. Data augmentation is Google SpecAugment with time warping and frequency mask and time mask to the spectrum on clean data set and noisy data set. TDNN and CNN model the spectrogram features from the time and space dimensions. RNN-type networks include RNN, Long Short-Term Memory network (LSTM), Gated Recurrent Unit (GRU), Bidirectional Long Short-Term Memory network (BiLSTM), and Bidirectional Gated Recurrent Unit (BiGRU). The RNN extracts hidden layer features and transforms them into high-level representations. The attention mechanism is selected to generate different weights and multiplied by the high-level representation generated by the RNN to obtain a fixed-length vector. Finally, we use a linear transformation and softmax function to generate scores. We also explored the size of attention mechanism, two attention mechanisms, rectified linear unit and hidden layer of RNN. Our model has achieved a true positive rate of 99.81% at a 5% false positive rate.

Isolated Keyword Spotting in Multilingual Environment using ANN and MFCC

The performance and analysis of Keyword Spotting system (KWS) are applied when the training and testing in a multilingual environment. This paper exhibits an approach for building up a multilingual KWS framework for Assamese, English and Hindi language dependent on feed-forward neural system. Mel Frequency Cepstral Coefficient (MFCC) has been utilized for highlight extraction which gives a lot of highlight vectors from recorded sound examples. Neural Network backpropagation model is utilized to improve the acknowledgment execution on the recently made multilingual database utilizing the multi-layer feed-forward neural system classifier.