Keyword Spotting Research Articles

Phone Synchronous Speech Recognition With CTC Lattices

Connectionist temporal classification CTC has recently shown improved performance and efficiency in automatic speech recognition. One popular decoding implementation is to use a CTC model to predict the phone posteriors at each frame and then perform Viterbi beam search on a modified WFST network. This is still within the traditional frame synchronous decoding framework. In this paper, the peaky posterior property of CTC is carefully investigated and it is found that ignoring blank frames will not introduce additional search errors. Based on this phenomenon, a novel phone synchronous decoding framework is proposed by removing tremendous search redundancy due to blank frames, which results in significant search speed up. The framework naturally leads to an extremely compact phone-level acoustic space representation: CTC lattice. With CTC lattice, efficient and effective modular speech recognition approaches, second pass rescoring for large vocabulary continuous speech recognition LVCSR, and phone-based keyword spotting KWS, are also proposed in this paper. Experiments showed that phone synchronous decoding can achieve 3-4 times search speed up without performance degradation compared to frame synchronous decoding. Modular LVCSR with CTC lattice can achieve further WER improvement. KWS with CTC lattice not only achieved significant equal error rate improvement, but also greatly reduced the KWS model size and increased the search speed.

Keyword spotting in handwritten chinese documents using semi-markov conditional random fields

This paper proposes a document indexing method for keyword spotting based on semi-Markov conditional random fields (semi-CRFs), which provide a theoretical framework for fusing the information of different contexts. The candidate segmentation-recognition lattice is first augmented based on the linguistic context to improve recognition results. For fast retrieval and to save storage space, the lattice is then purged by a forward-backward pruning procedure. In the reduced lattice, we estimate character similarity scores based on the semi-CRF model. The parameters of semi-CRF model are estimated using a binary classification objective, i.e., the cross-entropy (CE) to discriminate candidate characters in the lattice. To locate mis-recognized character instances in the lattice, we use confusing similar characters as proxies and search for proxy-characters in the index file. The proxy-character driven search can significantly improve the performance compared with our previous character-synchronous dynamic search (CSDS) method. Experimental results on the online handwriting database CASIA-OLHWDB justify the effectiveness of the proposed method.

HMM word graph based keyword spotting in handwritten document images

Line-level keyword spotting (KWS) is presented on the basis of frame-level word posterior probabilities. These posteriors are obtained using word graphs derived from the recognition process of a full-fledged handwritten text recognizer based on hidden Markov models and N-gram language models. This approach has several advantages. First, since it uses a holistic, segmentation-free technology, it does not require any kind of word or character segmentation. Second, the use of language models allows the context of each spotted word to be taken into account, thereby considerably increasing KWS accuracy. And third, the proposed KWS scores are based on true posterior probabilities, taking into account all (or most) possible word segmentations of the input image. These scores are properly bounded and normalized. This mathematically clean formulation lends itself to smooth, threshold-based keyword queries which, in turn, permit comfortable trade-offs between search precision and recall. Experiments are carried out on several historic collections of handwritten text images, as well as a well-known data set of modern English handwritten text. According to the empirical results, the proposed approach achieves KWS results comparable to those obtained with the recently-introduced “BLSTM neural networks KWS” approach and clearly outperform the popular, state-of-the-art “Filler HMM” KWS method. Overall, the results clearly support all the above-claimed advantages of the proposed approach.

Bayesian background models for keyword spotting in handwritten documents

Background in a handwritten document can be anything other than the words we are interested in. The characteristics of the background are typically captured by a background model to achieve spotting in handwritten documents. We propose two such Bayesian background models for keyword spotting in handwritten documents. Firstly, we present a background model using the Bayesian generalized linear model called (VDBM) and secondly propose a Bayesian generalized kernel background model called BGKBM. Given a set of handwritten documents and a bunch of keyword and non-keyword scores, the models learn an efficient Bayesian rejection criteria to output the most confident keyword regions in the handwritten document. For the variational dynamic background model (VDBM) the inference of parameters is done using variational methods and for the Bayesian generalized kernel background model (BGKBM), the inference is done using a proposed Markov chain Monte Carlo (MCMC) approach. The models are built on top of the scores returned by a handwritten recognizer for keywords and non-keywords. The approach is recognition based and works at line level. The methods have been validated on publicly available IAM dataset and compared with other state of the art line level keyword spotting approaches.

Word graphs size impact on the performance of handwriting document applications

Two document processing applications are considered: computer-assisted transcription of text images (CATTI) and Keyword Spotting (KWS), for transcribing and indexing handwritten documents, respectively. Instead of working directly on the handwriting images, both of them employ meta-data structures called word graphs (WG), which are obtained using segmentation-free handwritten text recognition technology based on N-gram language models and hidden Markov models. A WG contains most of the relevant information of the original text (line) image required by CATTI and KWS but, if it is too large, the computational cost of generating and using it can become unafordable. Conversely, if it is too small, relevant information may be lost, leading to a reduction of CATTI or KWS performance. We study the trade-off between WG size and performance in terms of effectiveness and efficiency of CATTI and KWS. Results show that small, computationally cheap WGs can be used without loosing the excellent CATTI and KWS performance achieved with huge WGs.

A Novel Lip Descriptor for Audio-Visual Keyword Spotting Based on Adaptive Decision Fusion

Keyword spotting remains a challenge when applied to real-world environments with dramatically changing noise. In recent studies, audio-visual integration methods have demonstrated superiorities since visual speech is not influenced by acoustic noise. However, for visual speech recognition, individual utterance mannerisms can lead to confusion and false recognition. To solve this problem, a novel lip descriptor is presented involving both geometry-based and appearance-based features in this paper. Specifically, a set of geometry-based features is proposed based on an advanced facial landmark localization method. In order to obtain robust and discriminative representation, a spatiotemporal lip feature is put forward concerning similarities among textons and mapping the feature to intra-class subspace. Moreover, a parallel two-step keyword spotting strategy based on decision fusion is proposed in order to make the best use of audio-visual speech and adapt to diverse noise conditions. Weights generated using a neural network combine acoustic and visual contributions. Experimental results on the OuluVS dataset and PKU-AV dataset demonstrate that the proposed lip descriptor shows competitive performance compared to the state of the art. Additionally, the proposed audio-visual keyword spotting (AV-KWS) method based on decision-level fusion significantly improves the noise robustness and attains better performance than feature-level fusion, which is also capable of adapting to various noisy conditions.

Querying out-of-vocabulary words in lexicon-based keyword spotting

Lexicon-based handwritten text keyword spotting (KWS) has proven to be a faster and more accurate alternative to lexicon-free methods. Nevertheless, since lexicon-based KWS relies on a predefined vocabulary, fixed in the training phase, it does not support queries involving out-of-vocabulary (OOV) keywords. In this paper, we outline previous work aimed at solving this problem and present a new approach based on smoothing the (null) scores of OOV keywords by means of the information provided by “similar” in-vocabulary words. Good results achieved using this approach are compared with previously published alternatives on different data sets.

Cross-language phoneme mapping for phonetic search keyword spotting using multiple source languages

Performing Phonetic Search Keyword Spotting (PS KWS) in new languages when language resources are scarce is an interesting and challenging task. In a previous paper we reported a methodology that enabled PS KWS under these conditions utilizing cross-language phoneme mappings from another sufficiently resourced and well-trained source language. We performed phoneme recognition in the new target language with the acoustic model of the source language. The keyword search was performed over a phoneme lattice of the target language phonemes following a mapping from one language to the other. In the present work we extend this method and its capabilities by mapping two source language phoneme sets into one target language set and performing a combined lattice search. Testing the technique on English and Arabic as source languages yielded a 50% Detection Rate (DR) and a False Alarm Rate (FAR - measured in number of false alarms per hour per keyword) of 2 when Spanish was the target language, a DR of 36% and FAR of 4 when Dari was the target language and a DR of 35% and FAR of 6 with Farsi as the target language. These results indicate that combining two source languages is better than using a single language since the acoustic space is better represented. Searching in a combined lattice while employing adequate phoneme transformations significantly improves performance. Such a system can be used as an initial version of a PS KWS system in a new language when sufficient language resources are not available.

A scalable pattern spotting system for historical documents

Information retrieval in historical documents has long consisted in spotting words. In this paper, we focus on graphical pattern spotting. Contrary to object detection and classification, where models of the object of interest may be trained, pattern spotting does not rely on any prior information regarding the query, nor predefined class of graphical objects. An offline sliding window approach may be suitable, provided that the challenge raised by high computational and storage costs is handled. We propose an unsupervised, segmentation-free approach that takes advantage of recent developments in computer vision to overcome these issues. We also investigate the use of new, compact descriptors for the data, namely the vectors of locally aggregated descriptors (VLAD) and Fisher Vectors, instead of the usual bag-of-visual-words approach. Results obtained on medieval manuscripts from the DocExplore project show that our approach achieves better retrieval results, with a better efficiency in terms of time/memory, compared to standard approaches. Experimentations show that VLAD and Fisher Vectors can be fruitfully used in the future for the description of historical documents. Additionally, we show that our system can be easily turned into a word spotting system with slight adaptation, and that it achieves results comparable to those recently published in ICDAR 2015 keyword spotting challenge.

Dialogue enabling speech-to-text user assistive agent system for hearing-impaired person.

A novel approach for assisting bidirectional communication between people of normal hearing and hearing-impaired is presented. While the existing hearing-impaired assistive devices such as hearing aids and cochlear implants are vulnerable in extreme noise conditions or post-surgery side effects, the proposed concept is an alternative approach wherein spoken dialogue is achieved by means of employing a robust speech recognition technique which takes into consideration of noisy environmental factors without any attachment into human body. The proposed system is a portable device with an acoustic beamformer for directional noise reduction and capable of performing speech-to-text transcription function, which adopts a keyword spotting method. It is also equipped with an optimized user interface for hearing-impaired people, rendering intuitive and natural device usage with diverse domain contexts. The relevant experimental results confirm that the proposed interface design is feasible for realizing an effective and efficient intelligent agent for hearing-impaired.

Decision Support System for Keyword Spotting Using Theory of Evidence

Keyword Spotting (KWS) in continuous speech is an emerging but challenging task that needs to deal with speech dynamics. Literature contains a variety of approaches for keyword spotting and spoken term detection. Most of these techniques are based on pattern based methods that are limited to a specific vocabulary of words with high computational cost for system training. A valuable contribution to the existing speaker dependent keyword spotting approaches is made in this paper by introducing a template matching based approach that exploits the Dempster's theory of Combined Evidence (CEv). The CEv plays a significant role as a Decision Support System (DSS) by integrating the keyword match/mismatch beliefs from multiple resources and providing a combined score (belief). Finally, performance of the proposed approach is compared with the existing keyword spotting techniques by using statistical analysis of the experimental results.

A Russian Keyword Spotting System Based on Large Vocabulary Continuous Speech Recognition and Linguistic Knowledge

The paper describes the key concepts of a word spotting system for Russian based on large vocabulary continuous speech recognition. Key algorithms and system settings are described, including the pronunciation variation algorithm, and the experimental results on the real-life telecom data are provided. The description of system architecture and the user interface is provided. The system is based on CMU Sphinx open-source speech recognition platform and on the linguistic models and algorithms developed by Speech Drive LLC. The effective combination of baseline statistic methods, real-world training data, and the intensive use of linguistic knowledge led to a quality result applicable to industrial use.

Investigation of DNN-Based Keyword Spotting in Low Resource Environments

Investigation of DNN-Based Keyword Spotting in Low Resource Environments

A fast hierarchical search algorithm for discriminative keyword spotting

A keyword spotter can be considered as a binary classifier which classifies a set of uttered sentences into two groups on the basis of whether they contain target keywords or not. For this classification task, the keyword spotter needs to identify the target keywords locations based on a fast and accurate search algorithm. In our previous works, we exploited a modified Viterbi (M-Viterbi) search algorithm which has two known drawbacks. First, to locate the target keywords, it runs an exhaustive search through all possible segments of input speech. Second, while computing the start and end time-frames of each new phone, it makes the keyword spotter to trace-back and re-evaluate the timing alignments of all previous one(s), despite the fact that very limited amount of data -if any- would get updated as a result. These two pitfalls cause a dramatically enlarged search space as well as a significant increase in computational complexity. In this paper, we propose a Hierarchical Search (H-Search) algorithm which allows the system to ignore some segments of input speech at each level of hierarchy, according to their lower likelihood of containing the target keywords. In addition, unlike the M-Viterbi algorithm, the H-Search algorithm does not demand repeated evaluations when computing the current phone alignment which, in turn, results in a narrowed-down search space (O(TP) versus O(TPLmax) – where T is number of frames, P is number of keyword phones and Lmax is the maximum phone duration) as well as a decreased computational complexity (O(TPLmax) versus O(TPLmax3)) compared to those of the M-Viterbi algorithm. We applied the H-Search algorithm to the classification part of an Evolutionary Discriminative Keyword Spotting (EDKWS) system introduced in our previous works. The experimental results indicate that the H-Search algorithm is executed 100 times faster than the M-Viterbi algorithm while the performance of the EDKWS system degrades no more than two percent compared to that of the M-Viterbi algorithm.

Assisted keyword indexing for lecture videos using unsupervised keyword spotting

Many students use videos to supplement learning outside the classroom. This is particularly important for students with challenged visual capacities, for whom seeing the board during lecture is difficult. For these students, we believe that recording the lectures they attend and providing effective video indexing and search tools will make it easier for them to learn course subject matter at their own pace. As a first step in this direction, we seek to help instructors create an index for their lecture videos using audio keyword search, with queries recorded by the instructor on their laptop and/or created from video excerpts. For this we have created an unsupervised within-speaker keyword spotting system. We represent audio data using de-noised, whitened and scale-normalized Mel Frequency Cepstral Coefficient (MFCC) features, and locate queries using Segmental Dynamic Time Warping (SDTW) of feature sequences. Our system is evaluated using introductory Linear Algebra lectures from instructors with different accents at two U.S. universities. For lectures produced using a video camera at RIT, laptop-recorded queries obtain an average Precision at 10 of 71.5%, while 79.5% is obtained for within-lecture queries. For lectures recorded using a lapel microphone at MIT, using a similar keyword set we obtain a much higher average Precision at 10 of 89.5%. Our results suggest that our system is robust to changes in environment, speaker and recording setup.

A hybrid model for automatic identification of risk factors for heart disease

Coronary artery disease (CAD) is the leading cause of death in both the UK and worldwide. The detection of related risk factors and tracking their progress over time is of great importance for early prevention and treatment of CAD. This paper describes an information extraction system that was developed to automatically identify risk factors for heart disease in medical records while the authors participated in the 2014 i2b2/UTHealth NLP Challenge. Our approaches rely on several nature language processing (NLP) techniques such as machine learning, rule-based methods, and dictionary-based keyword spotting to cope with complicated clinical contexts inherent in a wide variety of risk factors. Our system achieved encouraging performance on the challenge test data with an overall micro-averaged F-measure of 0.915, which was competitive to the best system (F-measure of 0.927) of this challenge task.

Cross-language phoneme mapping for phonetic search keyword spotting in continuous speech of under-resourced languages

As automatic speech recognition-based applications become increasingly common in a wide variety of market segments, thereis a growing need to support more languages. However, for many languages, the language resources needed to train speechrecognition engines are either limited or completely non-existent, and the process of acquiring or constructing new languageresources is both long and costly. This paper suggests a methodology that enables Phonetic Search Keyword Spotting to beimplemented in a large speech database of any given under-resourced language using cross-language phoneme mappings toanother language. The phoneme mapping enables a speech recognition engine from a sufficiently resourced and well-trainedsource language to be used for phoneme recognition in the new target language. The keyword search is then performed overa lattice of target language phonemes. Three cross-language phoneme mapping techniques are examined: knowledge-based,data-driven and phoneme recognition performance-based. The results suggest that Phonetic Search Keyword Spotting basedon the cross-language phoneme mapping approach proposed herein can serve as a quick initial solution for validating keywordspotting applications in new, under-resourced languages.

A New SVM Kernel for Keyword Spotting Using Confidence Measures

In this paper, we propose an alternative keyword spotting method relying on confidence measures and support vector machines. Confidence measures are computed from phone information provided by a Hidden Markov Model based speech recognizer. We use three kinds of techniques, i.e., arithmetic, geometric and harmonic means to compute a confidence measure for each word. The acceptance/rejection decision of a word is based on the confidence vector processed by the SVM classifier for which we propose a new Beta kernel. The performance of the proposed SVM classifier is compared with spotting methods based on some confidence means. Experimental results presented in this paper show that the proposed SVM classifier method improves the performances of the keyword spotting system.

Integrating natural language processing with image document analysis: what we learned from two real-world applications

Automatically accessing information from unconstrained image documents has important applications in business and government operations. These real-world applications typically combine optical character recognition (OCR) with language and information technologies, such as machine translation (MT) and keyword spotting. OCR output has errors and presents unique challenges to late-stage processing. This paper addresses two of these challenges: (1) translating the output from Arabic handwriting OCR which lacks reliable sentence boundary markers, and (2) searching named entities which do not exist in the OCR vocabulary, therefore, completely missing from Arabic handwriting OCR output. We address these challenges by leveraging natural language processing technologies, specifically conditional random field-based sentence boundary detection and out-of-vocabulary (OOV) name detection. This approach significantly improves our state-of-the-art MT system and achieves MT scores close to that achieved by human segmentation. The output from OOV name detection was used as a novel feature for discriminative reranking, which significantly reduced the false alarm rate of OOV name search on OCR output. Our experiments also show substantial performance gains from integrating a variety of features from multiple resources, such as linguistic analysis, image layout analysis, and image text recognition.

A study of Bag-of-Visual-Words representations for handwritten keyword spotting

The Bag-of-Visual-Words (BoVW) framework has gained popularity among the document image analysis community, specifically as a representation of handwritten words for recognition or spotting purposes. Although in the computer vision field the BoVW method has been greatly improved, most of the approaches in the document image analysis domain still rely on the basic implementation of the BoVW method disregarding such latest refinements. In this paper, we present a review of those improvements and its application to the keyword spotting task. We thoroughly evaluate their impact against a baseline system in the well-known George Washington dataset and compare the obtained results against nine state-of-the-art keyword spotting methods. In addition, we also compare both the baseline and improved systems with the methods presented at the Handwritten Keyword Spotting Competition 2014.