Discriminative Training Method Research Articles

A Novel Research in Low Altitude Acoustic Target Recognition Based on HMM

This paper introduces an improved HMM (hidden Markov model) for low altitude acoustic target recognition. To overcome the limitation of the classical CDHMM (continuous density hidden Markov model) training algorithm and the generalization ability deficiency of existing discriminative learning methods, a new discriminative training method for estimating the CDHMM in acoustic target recognition is proposed based on the principle of maximizing the minimum relative separation margin. According to the definition of the relative margin, the new training criterion can be equation as a standard constrained minimax optimization problem. Then, the optimization problem can be solved by a GPD (generalized probabilistic descent) algorithm. The experimental results show that the performance of the algorithm is significantly improved compared with the former training method, which can effectively improve the recognition ability of the acoustic target recognition system.

Effectiveness of dereverberation, feature transformation, discriminative training methods, and system combination approach for various reverberant environments

The recently released REverberant Voice Enhancement and Recognition Benchmark (REVERB) challenge includes a reverberant automatic speech recognition (ASR) task. This paper describes our proposed system based on multi-channel speech enhancement preprocessing and state-of-the-art ASR techniques. For preprocessing, we propose a single-channel dereverberation method with reverberation time estimation, which is combined with multichannel beamforming that enhances direct sound compared with the reflected sound. In addition, this paper also focuses on state-of-the-art ASR techniques such as discriminative training of acoustic models including the Gaussian mixture model, subspace Gaussian mixture model, and deep neural networks, as well as various feature transformation techniques. Although, for the REVERB challenge, it is necessary to handle various acoustic environments, a single ASR system tends to be overly tuned for a specific environment, which degrades the performance in the mismatch environments. To overcome this mismatch problem with a single ASR system, we use a system combination approach using multiple ASR systems with different features and different model types because a combination of various systems that have different error patterns is beneficial. In particular, we use our discriminative training technique for system combination that achieves better generalization by making systems complementary with the modified discriminative criteria. Experiments show the effectiveness of these approaches, reaching 6.76 and 18.60 % word error rates on the REVERB simulated and real test sets. These are 68.8 and 61.5 % relative improvements over the baseline.

Enhancer modeling uncovers transcriptional signatures of individual cardiac cell states in Drosophila.

Here we used discriminative training methods to uncover the chromatin, transcription factor (TF) binding and sequence features of enhancers underlying gene expression in individual cardiac cells. We used machine learning with TF motifs and ChIP data for a core set of cardiogenic TFs and histone modifications to classify Drosophila cell-type-specific cardiac enhancer activity. We show that the classifier models can be used to predict cardiac cell subtype cis-regulatory activities. Associating the predicted enhancers with an expression atlas of cardiac genes further uncovered clusters of genes with transcription and function limited to individual cardiac cell subtypes. Further, the cell-specific enhancer models revealed chromatin, TF binding and sequence features that distinguish enhancer activities in distinct subsets of heart cells. Collectively, our results show that computational modeling combined with empirical testing provides a powerful platform to uncover the enhancers, TF motifs and gene expression profiles which characterize individual cardiac cell fates.

Abstract 348: Multi-Species Genome-Wide Analyses of the Specification of Individual Cardiac Cell Fates

There are remarkable molecular and embryological similarities in cardiogenesis between Drosophila and vertebrates. Cells comprising the Drosophila heart can be subdivided into individual identities based on differences in morphology, function and gene expression patterns. Recent studies have shown that differential modifications of histone proteins, in vivo transcription factor (TF) binding, and the presence of particular TF binding motifs can be used as predictive signatures of the enhancers that govern cell-specific gene expression. Here we used discriminative training methods within an integrative, multi-species framework to uncover the motifs, enhancers and genes underlying cardiac cell fate decisions. As an initial step, we undertook a large-scale validation of Drosophila heart enhancers, which revealed enhancer activities in distinct subpopulations of cardiac cells. To identify related cell-specific regulatory elements, we used the validated enhancers as a training set in a machine learning approach that integrated TF motifs with ChIP data for both TF binding and histone modifications. Empirical validation of candidate enhancers predicted by this method confirmed activity in the appropriate cardiac cells. By clustering the motifs derived from the individual cardiac classifiers, we identified and validated sequence features which discriminate specific cellular identities. Next, we asked if similar predictive signatures underlie mouse and human cardiomyocyte (CM) differentiation from embryonic stem cells (ESCs). We show that the distribution of histone marks found within differentiating human and mouse ESCs indeed predict genes potentially critical for CM differentiation, with the best predictions provided by the overlapping mouse and human candidates. We evaluated this result in a large-scale RNAi-based screen of Drosophila orthologs of the mammalian genes, which uncovered dozens of novel cardiogenic regulators whose function is being tested in differentiating human ESCs. In total, these results document the utility of computational modeling combined with empirical testing to uncover the enhancers, TF motifs and genes which characterize individual cardiac cell fates in both invertebrate and mammalian species.

A comparative study of RPCL and MCE based discriminative training methods for LVCSR

This paper presents a comparative study of two discriminative methods, i.e., Rival Penalized Competitive Learning (RPCL) and Minimum Classification Error (MCE), for the tasks of Large Vocabulary Continuous Speech Recognition (LVCSR). MCE aims at minimizing a smoothed sentence error on training data, while RPCL focuses on avoiding misclassification through enforcing the learning of correct class and de-learning its best rival class. For a fair comparison, both the two discriminative mechanisms are implemented at the levels of phones and/or hidden Markov states using the same training corpus. The results show that both the MCE and RPCL based methods perform better than the Maximum Likelihood Estimation (MLE) based method. Comparing with the MCE based method, the RPCL based methods have better discriminative and generalizing abilities on both two levels.

Structured Adaptive Regularization of Weight Vectors for a Robust Grapheme-to-Phoneme Conversion Model

Grapheme-to-phoneme (g2p) conversion, used to estimate the pronunciations of out-of-vocabulary (OOV) words, is a highly important part of recognition systems, as well as text-to-speech systems. The current state-of-the-art approach in g2p conversion is structured learning based on the Margin Infused Relaxed Algorithm (MIRA), which is an online discriminative training method for multiclass classification. However, it is known that the aggressive weight update method of MIRA is prone to overfitting, even if the current example is an outlier or noisy. Adaptive Regularization of Weight Vectors (AROW) has been proposed to resolve this problem for binary classification. In addition, AROW’s update rule is simpler and more efficient than that of MIRA, allowing for more efficient training. Although AROW has these advantages, it has not been applied to g2p conversion yet. In this paper, we first apply AROW on g2p conversion task which is structured learning problem. In an evaluation that employed a dataset generated from the collective knowledge on the Web, our proposed approach achieves a 6.8% error reduction rate compared to MIRA in terms of phoneme error rate. Also the learning time of our proposed approach was shorter than that of MIRA in almost datasets. key words: g2p conversion, out-of-vocabulary word, online discriminative training, structured learning, AROW

Robust and Efficient Pattern Classification using Large Geometric Margin Minimum Classification Error Training

Recently, one of the standard discriminative training methods for pattern classifier design, i.e., Minimum Classification Error (MCE) training, has been revised, and its new version is called Large Geometric Margin Minimum Classification Error (LGM-MCE) training. It is formulated by replacing a conventional misclassification measure, which is equivalent to the so-called functional margin, with a geometric margin that represents the geometric distance between an estimated class boundary and its closest training pattern sample. It seeks the status of the trainable classifier parameters that simultaneously correspond to the minimum of the empirical average classification error count loss and the maximum of the geometric margin. Experimental evaluations showed the fundamental utility of LGM-MCE training. However, to increase its effectiveness, this new training required careful setting for hyperparameters, especially the smoothness degree of the smooth classification error count loss. Exploring the smoothness degree usually requires many trial-and-error repetitions of training and testing, and such burdensome repetition does not necessarily lead to an optimal smoothness setting. To alleviate this problem and further increase the effect of geometric margin employment, we apply in this paper a new idea that automatically determines the loss smoothness of LGM-MCE training. We first introduce a new formalization of it using the Parzen estimation of error count risk and formalize LGM-MCE training that incorporates a mechanism of automatic loss smoothness determination. Importantly, the geometric-margin-based misclassification measure adopted in LGM-MCE training is directly linked with the geometric margin in a pattern sample space. Based on this relation, we also prove that loss smoothness affects the production of virtual samples along the estimated class boundaries in pattern sample space. Finally, through experimental evaluations and in comparisons with other training methods, we elaborate the characteristics of LGM-MCE training and its new function that automatically determines an appropriate loss smoothness degree.

Discriminative Training for Automatic Speech Recognition: Modeling, Criteria, Optimization, Implementation, and Performance

Discriminative training techniques have been shown to consistently outperform the maximum likelihood (ML) paradigm for acoustic model training in automatic speech recognition (ASR). Consequently, today's discriminative training methods are fundamental components of state-of-the-art systems and are a major line of research in speech recognition. This article gives a comprehensive overview of discriminative training methods for acoustic model training in the context of ASR. The article covers all related aspects of discriminative training for speech recognition, i.e., specific training criteria and their relation, statistical modeling, different parameter optimization approaches, efficient implementation of discriminative training, and a performance overview.

Broad phonetic class definition driven by phone confusions

Intermediate representations between the speech signal and phones may be used to improve discrimination among phones that are often confused. These representations are usually found according to broad phonetic classes, which are defined by a phonetician. This article proposes an alternative data-driven method to generate these classes. Phone confusion information from the analysis of the output of a phone recognition system is used to find clusters at high risk of mutual confusion. A metric is defined to compute the distance between phones. The results, using TIMIT data, show that the proposed confusion-driven phone clustering method is an attractive alternative to the approaches based on human knowledge. A hierarchical classification structure to improve phone recognition is also proposed using a discriminative weight training method. Experiments show improvements in phone recognition on the TIMIT database compared to a baseline system.

Improvement Comparison of Different Lattice-based Discriminative Training Methods in Chinese-monolingual and Chinese-English-bilingual Speech Recognition

AbstractDiscriminative training approaches such as minimum phone error (MPE), feature minimum phone error (fMPE) and boosted maximum mutual information (BMMI) have brought remarkable improvement to the speech community in recent years, however, much work still remains to be done. This paper investigates the performances of three lattice-based discriminative training methods in detail, and does a comparison of different I-smoothing methods to obtain more robust models in the Chinese-monolingual situation. The complementary properties of the different discriminative training methods are explored to perform a system combination by recognizer output voting error reduction (ROVER). Although discriminative training is normally used in monolingual systems, this paper systematically investigates its use for bilingual speech recognition, including MPE, fMPE, and BMMI. A new method is proposed to generate significantly better lattices for training the bilingual model, and complementary discriminative training models are also explored to get the best ROVER performance in the bilingual situation. Experimental results show that all forms of discriminative training can reduce the word error rate in both monolingual and bilingual systems, and that combining complementary discriminative training methods can improve the performance significantly.

Round-Robin Duel Discriminative Language Models

Discriminative training has received a lot of attention from both the machine learning and speech recognition communities. The idea behind the discriminative approach is to construct a model that distinguishes correct samples from incorrect samples, while the conventional generative approach estimates the distributions of correct samples. We propose a novel discriminative training method and apply it to a language model for reranking speech recognition hypotheses. Our proposed method has round-robin duel discrimination (R2D2) criteria in which all the pairs of sentence hypotheses including pairs of incorrect sentences are distinguished from each other, taking their error rate into account. Since the objective function is convex, the global optimum can be found through a normal parameter estimation method such as the quasi-Newton method. Furthermore, the proposed method is an expansion of the global conditional log-linear model whose objective function corresponds to the conditional random fields. Our experimental results show that R2D2 outperforms conventional methods in many situations, including different languages, different feature constructions and different difficulties.

Improvement Comparison of Different Lattice-based Discriminative Training Methods in Chinese-monolingual and Chinese-English-bilingual Speech Recognition

Improvement Comparison of Different Lattice-based Discriminative Training Methods in Chinese-monolingual and Chinese-English-bilingual Speech Recognition

Discriminative Training for Multiple Observation Likelihood Ratio Based Voice Activity Detection

It is possible to show that the likelihood ratio (LR) test from multiple observations can enhance the performance of a statically modeled voice actively detection (VAD) system. However, the combination weights for the likelihood ratios (LRs) in each observation are rather empirical and heuristical. In this study, the optimal combination weights from two discriminative training methods are studied to directly improve VAD performance, in terms of reduced misclassification errors and improved receiver operating characteristics (ROC) curves. As shown in the evaluations, VAD performance, both in terms of absolute performance and consistency across noise types, can be significantly improved using the proposed method.

Minimum classification error learning for sequential data in the wavelet domain

Wavelet analysis has found widespread use in signal processing and many classification tasks. Nevertheless, its use in dynamic pattern recognition have been much more restricted since most of wavelet models cannot handle variable length sequences properly. Recently, composite hidden Markov models which observe structured data in the wavelet domain were proposed to deal with this kind of sequences. In these models, hidden Markov trees account for local dynamics in a multiresolution framework, while standard hidden Markov models capture longer correlations in time. Despite these models have shown promising results in simple applications, only generative approaches have been used so far for parameter estimation. The goal of this work is to take a step forward in the development of dynamic pattern recognizers using wavelet features by introducing a new discriminative training method for this Markov models. The learning strategy relies on the minimum classification error approach and provides re-estimation formulas for fully non-tied models. Numerical experiments on phoneme recognition show important improvement over the recognition rate achieved by the same models trained using maximum likelihood estimation.

Minimum Classification Error Training of Hidden Markov Models for Sequential Data in the Wavelet Domain

In the last years there has been increasing interest in developing discriminative training methods for hidden Markov models, with the aim to improve their performance in classi cation and pattern recognition tasks. Although several advances have been made in this area, they have been targeted almost exclusively to standard models whose conditional observations are given by a Gaussian mixture density. In parallel with this development, a special kind of hidden Markov models de ned in the wavelet domain has found wide-spread use in the signal and image processing community. Nevertheless, these models have been typically restricted to fully-tied parameter training using a single sequence and maximum likelihood estimates. This paper takes a step forward in the development of sequential pattern recognizers based on wavelet-domain hidden Markov models by introducing a new discriminative training method. The learning strategy relies on the minimum classi cation error approach and provides reestimation formulas for fully non-tied models. Numerical experiments on a simple phoneme recognition task show important improvement over the recognition rate achieved by the same models trained under the maximum likelihood estimation approach.

Discriminative Parameter Training of Unscented Kalman Filter

In this paper, we propose a learning technique to solve parameter adjustment problems of Unscented Kalman Filter (UKF) for accurate localization. The parameters of UKF consist of three kinds of parameters as follows: I) the covariance matrix of input noise, II) the covariance matrix of measurement noise, III) Hyper-parameters of UKF. UKF localization performance critically depends on these parameters. One of discriminative training methods is adopted to obtain the optimal parameters. A coordinate ascent algorithm is used as a training algorithm to optimize these parameters included in complicated functions. Simulation and experimental results are presented. The localization result using the proposed method outperforms the result of hand tuning and the result of covariance learning.

Discriminative training of HMMs for automatic speech recognition: A survey

Recently, discriminative training (DT) methods have achieved tremendous progress in automatic speech recognition (ASR). In this survey article, all mainstream DT methods in speech recognition are reviewed from both theoretical and practical perspectives. From the theoretical aspect, many effective discriminative learning criteria in ASR are first introduced and then a unifying view is presented to elucidate the relationship among these popular DT criteria originally proposed from different viewpoints. Next, some key optimization methods used to optimize these criteria are summarized and their convergence properties are discussed. Moreover, as some recent advances, a novel discriminative learning framework is introduced as a general scheme to formulate discriminative training of HMMs for ASR, from which a variety of new DT methods can be developed. In addition, some important implementation issues regarding how to conduct DT for large vocabulary ASR are also discussed from a more practical aspect, such as efficient implementation of discriminative training on word graphs and effective optimization of complex DT objective functions in high-dimensionality space, and so on. Finally, this paper is summarized and concluded with some possible future research directions for this area. As a technical survey, all DT techniques and ideas are reviewed and discussed in this paper from high level without involving too much technical detail and experimental result.

Voice activity detection based on statistical models and machine learning approaches

The voice activity detectors (VADs) based on statistical models have shown impressive performances especially when fairly precise statistical models are employed. Moreover, the accuracy of the VAD utilizing statistical models can be significantly improved when machine-learning techniques are adopted to provide prior knowledge for speech characteristics. In the first part of this paper, we introduce a more accurate and flexible statistical model, the generalized gamma distribution (G ΓD) as a new model in the VAD based on the likelihood ratio test. In practice, parameter estimation algorithm based on maximum likelihood principle is also presented. Experimental results show that the VAD algorithm implemented based on G ΓD outperform those adopting the conventional Laplacian and Gamma distributions. In the second part of this paper, we introduce machine learning techniques such as a minimum classification error (MCE) and support vector machine (SVM) to exploit automatically prior knowledge obtained from the speech database, which can enhance the performance of the VAD. Firstly, we present a discriminative weight training method based on the MCE criterion. In this approach, the VAD decision rule becomes the geometric mean of optimally weighted likelihood ratios. Secondly, the SVM-based approach is introduced to assist the VAD based on statistical models. In this algorithm, the SVM efficiently classifies the input signal into two classes which are voice active and voice inactive regions with nonlinear boundary. Experimental results show that these training-based approaches can effectively enhance the performance of the VAD.

Improving the characterization of the alternative hypothesis via minimum verification error training with applications to speaker verification

Speaker verification is usually formulated as a statistical hypothesis testing problem and solved by a log-likelihood ratio (LLR) test. A speaker verification system's performance is highly dependent on modeling the target speaker's voice (the null hypothesis) and characterizing non-target speakers’ voices (the alternative hypothesis). However, since the alternative hypothesis involves unknown impostors, it is usually difficult to characterize a priori. In this paper, we propose a framework to better characterize the alternative hypothesis with the goal of optimally distinguishing the target speaker from impostors. The proposed framework is built on a weighted arithmetic combination (WAC) or a weighted geometric combination (WGC) of useful information extracted from a set of pre-trained background models. The parameters associated with WAC or WGC are then optimized using two discriminative training methods, namely, the minimum verification error (MVE) training method and the proposed evolutionary MVE (EMVE) training method, such that both the false acceptance probability and the false rejection probability are minimized. Our experiment results show that the proposed framework outperforms conventional LLR-based approaches.

Conrad: Gene prediction using conditional random fields

We present Conrad, the first comparative gene predictor based on semi-Markov conditional random fields (SMCRFs). Unlike the best standalone gene predictors, which are based on generalized hidden Markov models (GHMMs) and trained by maximum likelihood, Conrad is discriminatively trained to maximize annotation accuracy. In addition, unlike the best annotation pipelines, which rely on heuristic and ad hoc decision rules to combine standalone gene predictors with additional information such as ESTs and protein homology, Conrad encodes all sources of information as features and treats all features equally in the training and inference algorithms. Conrad outperforms the best standalone gene predictors in cross-validation and whole chromosome testing on two fungi with vastly different gene structures. The performance improvement arises from the SMCRF's discriminative training methods and their ability to easily incorporate diverse types of information by encoding them as feature functions. On Cryptococcus neoformans, configuring Conrad to reproduce the predictions of a two-species phylo-GHMM closely matches the performance of Twinscan. Enabling discriminative training increases performance, and adding new feature functions further increases performance, achieving a level of accuracy that is unprecedented for this organism. Similar results are obtained on Aspergillus nidulans comparing Conrad versus Fgenesh. SMCRFs are a promising framework for gene prediction because of their highly modular nature, simplifying the process of designing and testing potential indicators of gene structure. Conrad's implementation of SMCRFs advances the state of the art in gene prediction in fungi and provides a robust platform for both current application and future research.