Keyword Spotting Research Articles

Isolated Keyword Spotting in Multilingual Environment using ANN and MFCC

The performance and analysis of Keyword Spotting system (KWS) are applied when the training and testing in a multilingual environment. This paper exhibits an approach for building up a multilingual KWS framework for Assamese, English and Hindi language dependent on feed-forward neural system. Mel Frequency Cepstral Coefficient (MFCC) has been utilized for highlight extraction which gives a lot of highlight vectors from recorded sound examples. Neural Network backpropagation model is utilized to improve the acknowledgment execution on the recently made multilingual database utilizing the multi-layer feed-forward neural system classifier.

Isolated Keyword Spotting in Multilingual Environment using ANN and MFCC

The performance and analysis of Keyword Spotting system (KWS) are applied when the training and testing in a multilingual environment. This paper exhibits an approach for building up a multilingual KWS framework for Assamese, English and Hindi language dependent on feed-forward neural system. Mel Frequency Cepstral Coefficient (MFCC) has been utilized for highlight extraction which gives a lot of highlight vectors from recorded sound examples. Neural Network backpropagation model is utilized to improve the acknowledgment execution on the recently made multilingual database utilizing the multi-layer feed-forward neural system classifier.

A multimodel keyword spotting system based on lip movement and speech features

The spoken keyword recognition and its localization are one of the fundamental aspects of speech recognition and known as keyword spotting. In automatic keyword spotting systems, the Lip-reading (LR) methods have a broader role when audio data is not present or has corrupted information. The available works from the literature have focussed on recognizing a limited number of words or phrases and require the cropped region of face or lip. Whereas the proposed model does not require the cropping of the video frames and it is recognition free. The proposed model is utilizing Convolutional Neural Networks and Long Short Term Memory networks to improve the overall performance. The model creates a 128-dimensional subspace to represent the feature vectors for speech signals and corresponding lip movements (focused viseme sequences). Thus the proposed model can tackle lip reading as an unconstrained natural speech signal in the video sequences. In the experiments, different standard datasets as LRW (Oxford-BBC), MIRACL-VC1, OuluVS, GRID, and CUAVE are used for the evaluation of the proposed model. The experiments also have a comparative analysis of the proposed model with current state-of-the-art methods for Lip-Reading task and keyword spotting task. The proposed model obtain excellent results for all datasets under consideration.

Vocell: A 65-nm Speech-Triggered Wake-Up SoC for 10-$\mu$ W Keyword Spotting and Speaker Verification

The use of speech-triggered wake-up interfaces has grown significantly in the last few years for use in ubiquitous and mobile devices. Since these interfaces must always be active, power consumption is one of their primary design metrics. This article presents a complete mixed-signal system-on-chip, capable of directly interfacing to an analog microphone and performing keyword spotting (KWS) and speaker verification (SV), without any need for further external accesses. Through the use of: 1) an integrated single-chip digital-friendly design; b) hardware-aware algorithmic optimization; and c) memory- and power-optimized accelerators, ultra-low power is achieved while maintaining high accuracy for speech recognition tasks. The 65-nm implementation achieves 18.3- $\mu \text{W}$ worst case power consumption or 10.6- $\mu \text{W}$ power for typical real-time scenarios, $10\times $ below state of the art (SoA).

Using keyword spotting systems as tools for the transcription of historical handwritten documents: Models and procedures for performance evaluation

The paper proposes a performance model for estimating the improvement of the time needed to transcribe small collections of handwritten documents by using a keyword spotting system (KWS) with respect to the time for manually achieving the transcription. The proposed model assumes that no other information than those obtained from the samples and the KWS system performance on the training set are available, and, depending on them, establishes analytically the condition the performance measures must satisfy to make it profitable to use the system, and, in the affirmative case, estimates the gain and the accuracy of such estimation. The model is complemented by a step-by-step procedure for building the training set, running the KWS on it, estimating the performance parameters on the data set, and eventually estimating the overall improvement and the accuracy of this estimate.

Improved External Speaker-Robust Keyword Spotting for Hearing Assistive Devices

For certain applications, keyword spotting (KWS) requires some degree of personalization. This is the case for KWS for hearing assistive devices, e.g., hearing aids, where only the device user should be allowed to trigger the KWS system. In this paper, we first develop a new realistic hearing aid experimental framework. Next, using this framework we show that the performance of a state-of-the-art multi-task deep learning architecture exploiting cepstral features for joint KWS and users’ own-voice/external speaker detection drops significantly. To overcome this problem, we use phase difference information through GCC-PHAT (Generalized Cross-Correlation with PHAse Transform)-based coefficients along with log-spectral magnitude features. In addition, we demonstrate that working in the perceptually-motivated constant-Q transform (CQT) domain instead of in the short-time Fourier transform (STFT) domain allows for the generation of compact and coherent features which provide superior KWS performance. Our experimental results show that our CQT-based proposal achieves a relative KWS accuracy improvement of around 18% compared to using cepstral features while dramatically decreasing the number of multiplications in the multi-task architecture, which is key in the context of low-resource devices like hearing assistive devices.

QCNN Inspired Reconfigurable Keyword Spotting Processor With Hybrid Data-Weight Reuse Methods

The keyword spotting (KWS) system is one of the most important interfaces between humans and machines since it is usually the start of automatic speech recognition and natural language processing techniques. However, for KWS hardware, it is still a problem to make one specified chip both low power and high performed under multiple scenarios, such as in meeting rooms, on different traffic or in parks and so on, for different scenarios own wide range signal-noise-ratios (SNRs). The problem leads to the requirements of balanced design between KWS system accuracy and the hardware cost under various noise types and levels. To overcome the balanced design and tradeoff problems, a complete KWS processor including an Mel-Frequency Cepstrum Coefficients (MFCC) feature extractor and a quantized Convolutional Neural Network (QCNN) accelerator is proposed for wide SNR range and low-power KWS in this paper. Firstly, the approach to quantize CNNs into QCNNs with high accuracy is proposed with considerations of hardware-software tradeoff. With the tradeoff of KWS system accuracy and hardware cost, the 4bit/8bit dual-working-mode strategy is proposed to keep low hardware cost and high accuracy under different scenarios. To be specific, the training, tuning and validating of the CNNs and QCNNs are taken with the dataset of 10 keywords chosen from the Google Command Speech Dataset (GCSD). Secondly, a serial FFT based MFCC extractor is implemented with low power and small footprint. Finally, with a novel hybrid reuse strategy of input data and network weight, a reconfigurable and approximate computing based QCNN accelerator is designed. Implemented and verified under TSMC 22nm ULL technology, with the area of 1.42mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , the QCNN accelerator can achieve 5.26μW/9.08μW power consumption in 4bit/8bit work mode with accuracy of 88% and 93% respectively, which is superior to the state-of-the-art processors.

KWA: A New Method of Calculation and Representation Accuracy for Speech Keyword Spotting in String Results

This study proposes a new method to measure and represent accuracy for Keyword Spotting (KWS) problem in non-aligned string results. Our approach, called Keyword Spotting Accuracy (KWA), was improved from the Levenshtein Distance algorithm, that used to evaluate the accuracy of the keywords in KWS by measuring the minimum distance between two strings. The main improved algorithm is to show the status of each keyword in training phase for predicted and true labels. In which, representing which words are correct, which ones need to be inserted, substituted or deleted when comparing the prediction labels with true ones during the training phase. In addition, a new method of presenting the multiple keywords in results was proposed to indicate the accuracy of each keyword. This method can display detailed results by keywords, from which, we can obtain the accuracy, distribution, and balance of the keywords in the training dataset by actual speech variance, not by counting keywords in true labels as usual.

Gradual Weight Updating for Sentiment Mining

Nowadays, many people prefer the use of social media for communicating and exchanging opinions with each other over face to face communication. This has lead to a generation of a tremendous amount of textual opinioned data. Understanding this opinioned data is useful from all perspectives. But the major challenge exists here is how to extract the exact sentiment hidden behind this huge data. To solve this problem, keyword spotting or dictionary-based approaches are followed. In this paper, we present a Gradual Weight Updating for sentiment mining. It not only considers the polarity of each word similar to the unigram methodology but, it also focuses on the entire cluster of words that contains the unigram. The different steps it follows for sentiment extraction of the word are polarity fetching, cluster marking, weight tagging, valence shifter, adversative conjunction handling, and final score generation. The paper contributions in the area of domain independent opinioned word extraction and accurate polarity mining with the help of context marking approach. We used the various opinionated datasets to compare and illustrate the performance of our proposed system.

Speech Sanitizer: Speech Content Desensitization and Voice Anonymization

Voice input users’ speech recordings are being collected by service providers and shared with third parties, who may abuse users’ voiceprints, identify them by voice, and learn their sensitive speech content. In this work, we design <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Speech Sanitizer</i> to perturb users’ speech recordings so that the sanitized speech can be safely shared with third parties. First, we desensitize speech content by identifying sensitive words, localizing them in the audio using DTW-based keyword spotting, and substituting them with safe words. Both common and personalized sensitive words are identified and replaced. Then, we anonymize users’ voiceprints with a carefully designed voice conversion mechanism that is resistant to de-anonymization attacks. Meanwhile, we try to preserve the utility of the sanitized speech, measured by the accuracy of speech recognition performed on it. We implement Speech Sanitizer and present extensive experimental results that validate the effectiveness and efficiency of our algorithms. It is demonstrated that we are able to reduce the chance of a user's voice being identified from 50 people by 83.7 percent while keeping the drop of speech recognition accuracy within 19.1 percent. We can also easily relax the privacy level to improve speech recognition accuracy.

Experimental research on the impact of similarity function selection on the quality of keyword spotting in speech signal

The paper describes an evaluation of the application of selected similarity functions in the task of keyword spotting. Experiments were carried out in the Polish language. The research results can be used to improve already existing keyword spotting methods, or to develop new ones.

A 3.89-GOPS/mW Scalable Recurrent Neural Network Processor With Improved Efficiency on Memory and Computation

Recurrent neural networks (RNNs) perform excellently on sequencing tasks but are severely restricted by the complex computations and intensive memory consumption due to their internal fully connected topologies, thereby making it a great challenge to implement RNNs on embedded devices. In this brief, we propose an energy-efficient RNN processor by exploiting the data locality in network compression using an innovative quantified sparse matrix encoding format. Compared with the conventional processors for compressed RNNs, more than 80% of the weight fetching and matrix–vector multiplications can be further reduced in applications, such as natural language and keyword spotting. To handle different scales of RNN models without introducing significant interactive overhead, scalable hardware architecture is presented to organize multiple processor engines in a spatial fashion with the assistance of the network cross-division strategy. Synthesized in the SMIC 40LL CMOS process, the prototype processor has a total area of 0.65 mm2 with 95.5 kB of static random-access memory capacity. Based on the simulation, this processor achieves a peak performance of 24 GOPS and dissipates 6.16-mW power with 1.1 V supply and 200 MHz. The peak energy efficiency reaches 3.89 GOPS/mW, which is state of the art among existing RNN accelerators.

An FPGA-Based Hardware Accelerator for CNNs Using On-Chip Memories Only: Design and Benchmarking with Intel Movidius Neural Compute Stick

During the last years, convolutional neural networks have been used for different applications, thanks to their potentiality to carry out tasks by using a reduced number of parameters when compared with other deep learning approaches. However, power consumption and memory footprint constraints, typical of on the edge and portable applications, usually collide with accuracy and latency requirements. For such reasons, commercial hardware accelerators have become popular, thanks to their architecture designed for the inference of general convolutional neural network models. Nevertheless, field-programmable gate arrays represent an interesting perspective since they offer the possibility to implement a hardware architecture tailored to a specific convolutional neural network model, with promising results in terms of latency and power consumption. In this article, we propose a full on-chip field-programmable gate array hardware accelerator for a separable convolutional neural network, which was designed for a keyword spotting application. We started from the model implemented in a previous work for the Intel Movidius Neural Compute Stick. For our goals, we appropriately quantized such a model through a bit-true simulation, and we realized a dedicated architecture exclusively using on-chip memories. A benchmark comparing the results on different field-programmable gate array families by Xilinx and Intel with the implementation on the Neural Compute Stick was realized. The analysis shows that better inference time and energy per inference results can be obtained with comparable accuracy at expenses of a higher design effort and development time through the FPGA solution.

Region Proposal Network Based Small-Footprint Keyword Spotting

We apply an anchor-based region proposal network (RPN) for end-to-end keyword spotting (KWS). RPNs have been widely used for object detection in image and video processing; here, it is used to jointly model keyword classification and localization. The method proposes several anchors as rough locations of the keyword in an utterance and jointly learns classification and transformation to the ground truth region for each positive anchor. Additionally, we extend the keyword/non-keyword binary classification to detect multiple keywords. We verify our proposed method on a hotword detection data set with two hotwords. At a false alarm rate of one per hour, our method achieved more than 15% relative reduction in false rejection of the two keywords over multiple recent baselines. In addition, our method predicts the location of the keyword with over 90% overlap, which can be important for many applications.

CTC token parsing algorithm using keyword spotting for BLSTM based unconstrained handwritten recognition

The images are becoming more popular in today’s world and being made available over the internet, scanned/captured documents are used in paperless offices and digital libraries. The keyword spotting is well known techniques in document image retrieval system and Recognition free Information retrieval is based on Keyword Spotting technique it searches for most related keyword from image as per user request by using only image features. In this paper, to avoid vanishing gradient problem in the handwritten recognition using parser methods and also character matching to perform efficient and reliable output result.

Exploiting Future Word Contexts in Neural Network Language Models for Speech Recognition

Language modeling is a crucial component in a wide range of applications including speech recognition. Language models (LMs) are usually constructed by splitting a sentence into words and computing the probability of a word based on its word history. This sentence probability calculation, making use of conditional probability distributions, assumes that there is little impact from approximations used in the LMs, including the word history representations and finite training data. This motivates examining models that make use of additional information from the sentence. In this paper, future word information, in addition to the history, is used to predict the probability of the current word. For recurrent neural network LMs (RNNLMs), this information can be encapsulated in a bi-directional model. However, if used directly, this form of model is computationally expensive when trained on large quantities of data, and can be problematic when used with word lattices. This paper proposes a novel neural network language model structure, the succeeding-word RNNLM, su-RNNLM, to address these issues. Instead of using a recurrent unit to capture the complete future word contexts, a feedforward unit is used to model a fixed finite number of succeeding words. This is more efficient in training than bi-directional models and can be applied to lattice rescoring. The generated lattices can be used for downstream applications, such as confusion network decoding and keyword search. Experimental results on speech recognition and keyword spotting tasks illustrate the empirical usefulness of future word information, and the flexibility of the proposed model to represent this information.

Hybrid HMM/BLSTM system for multi-script keyword spotting in printed and handwritten documents with identification stage

In this paper, we propose a novel script-independent approach for word spotting in printed and handwritten multi-script documents. Since each writing type and script need to be processed using a specific spotting engine, the proposed system proceeds on two stages. The script identification is a preliminary stage that aims at recognizing on one level the writing type and the script of the input image document. Second, a specific word spotting method is used to spot query words in a large collection of documents. The proposed spotting system is based on deep bidirectional long short-term memory neural network and hidden Markov model (HMM) hybrid architecture. It takes advantage of DNN’s strong representation learning power and HMM’s sequential modeling ability. The global system has been evaluated on a mixed corpus of public databases such as KHATT, PKHATT for Arabic script and RIMES for Latin script. The experimental results on script identification and keyword spotting confirm the effectiveness of the proposed approach.

A survey on structured discriminative spoken keyword spotting

Spoken keyword spotting refers to the detection of all occurrences of desired words in continuous speech utterances. This paper includes a comprehensive review on various spoken keyword spotting (especially discriminative spoken keyword spotting) approaches. The most common datasets and evaluation measures for training and evaluating the spoken keyword spotting systems are reviewed in this paper. Moreover, the main framework for structured discriminative spoken keyword spotting (SDKWS) is presented. Different parts of the SDKWS framework such as feature extraction, model training, search algorithm and thresholding are discussed in this paper. Finally, the paper is concluded in the conclusion section and the future works are presented in the last part of that section.

Adaptive spectral smoothening for development of robust keyword spotting system

It is well known that a keyword spotting (KWS) system provides significantly reduced performance in mismatched training and test conditions. In this work, an approach is proposed for reducing the mismatches between the training and test speech due to speaker-related variabilities and environmental noises. In the proposed approach, the variational-mode decomposition is first performed on the short-term magnitude spectra to decompose it into a number of variational mode functions (VMFs) in an adaptive manner. Then, a sufficiently smoothed spectra are reconstructed by selecting only two lower frequency VMFs. When the KWS system is developed by using Mel frequency cepstral coefficients (MFCCs) extracted from the smoothed spectra, a significantly improved performance is observed for pitch and noise mismatched test conditions. To further suppress the mismatches due to the pitch and speaking rate of the speakers, data-augmented training based on explicit prosody modification is performed. The experimental results presented in this study show that data-augmented training further enhances the performance of the developed KWS.

Word Spotting Based on Bispace Similarity for Visual Information Retrieval in Handwritten Document Images

Retrieving information from a huge collection of ancient handwritten documents is important for indexing, interpreting, browsing, and searching documents in various domains. Word spotting approaches are widely used in this context but have several limitations related to the complex properties of handwriting. These can appear at several steps: interest point detection, description, and matching. This article proposes a new word spotting approach for the word retrieval in handwritten document, which mainly leverages the properties of image gradients for visual features detection and description. The proposed approach is based on the combination of spatial relationships with textural information to design a more accurate matching. The experimental results of the proposed approach demonstrate a higher performance over the Jeremy Bentham dataset, evaluated following the recent benchmarks of ICDAR 2015 Competition on Keyword Spotting for Handwritten Documents.