Maximum Mutual Information Criterion Research Articles

Making up the shortages of the Bayes classifier by the maximum mutual information classifier

The Bayes classifier is often used because it is simple, and the maximum posterior probability (MPP) criterion it uses is equivalent to the least error rate criterion. However, it has issues in the following circumstances: (i) if information instead of correctness is more important, we should use the maximum likelihood criterion or maximum information criterion, which can reduce the rate of failure to report small probability events. (ii) For unseen instance classifications, the previously optimised classifier cannot be properly used when the probability distribution of true classes is changed. (iii) When classes’ feature distributions instead of transition probability functions (TPFs) are stable, it is improper to train the TPF, such as the logistic function, with parameters. (iv) For multi-label classifications, it is difficult to optimise a group of TPFs with parameters that the Bayes classifier needs. This study addresses these issues by comparing the MPP criterion with the maximum likelihood criterion and maximum mutual information (MMI) criterion. It suggests using the MMI criterion for most unseen instance classifications. It presents a new iterative algorithm, the channel matching (CM) algorithm, for the MMI classification. It uses two examples to show the advantages of the CM algorithm: fast and reliable.

End-to-End Amdo-Tibetan Speech Recognition Based on Knowledge Transfer

The end-to-end speech recognition technology solves the problem that each component is independent and models cannot be jointly optimized in the traditional speech recognition model. It incorporates such components as the acoustic model, language model, and decoding unit of the hybrid model into a single neural network, that can avoid the inherent defects of multiple modules and greatly reduces the complexity of the speech recognition model. In this research, an Amdo-Tibetan speech recognition system is constructed based on Listen, Attend and Spell (LAS) model by the end-to-end speech recognition technology. It can realize the direct conversion from Amdo-Tibetan speech sequence to the corresponding character sequence and greatly reduces the difficulty of building the Amdo-Tibetan speech recognition model. To further improve the performance of the proposed system, the following improvements have been made: firstly, the Multi-Head Attention mechanism is introduced to improve the alignment accuracy between state vectors of decoder and encoder; secondly, the label smoothing technique is adopted to solve the problem of over-fitting; thirdly, an N-gram language model is combined with the LAS model to increase the accuracy of speech recognition and the maximum mutual information (MMI) criterion is employed for discriminative training; and finally, transfer learning is utilized to overcome the problem of insufficient training data. Experimental results show that the proposed model can significantly enhance the performance of Amdo-Tibetan speech recognition.

A MIMO Radar-Based DOA Estimation Structure Using Compressive Measurements.

In this paper, we propose a novel direction-of-arrival (DOA) estimation structure based on multiple-input multiple-output (MIMO) radar with colocated antennas, referred to as compressive measurement-based MIMO (CM-MIMO) radar, where the compressive sensing (CS) is employed to reduce the number of channels. Therefore, the system complexity and the computational burden are effectively reduced. It is noted that CS is used after the matched filters and that a measurement matrix with less rows than columns is multiplied with the received signals. As a result, the configurations of the transmit and receive antenna arrays are not affected by the CS and can be determined according to the practical requirements. To study the estimation performance, the Cramér–Rao bound (CRB) with respect to the DOAs of the proposed CM-MIMO radar is analyzed in this paper. The derived CRB expression is also suitable for the conventional MIMO radar by setting the measurement matrix as an identity matrix. Moreover, the CRB expression can work in the under-determined case, since the sum-difference coarray structure is considered. However, the random measurement matrix leads to high information loss, thus compromising the estimation performance. To overcome this problem, we consider that the a prior probability distribution of the DOAs associated with the targets can be obtained in many scenarios and an optimization approach for the measurement matrix is proposed in this paper, where the maximum mutual information criterion is adopted. The superiority of the proposed structure is validated by numerical simulations.

Adaptive Waveform Design for Cognitive Radar in Multiple Targets Situation.

In this paper, the problem of cognitive radar (CR) waveform optimization design for target detection and estimation in multiple extended targets situations is investigated. This problem is analyzed in signal-dependent interference, as well as additive channel noise for extended targets with unknown target impulse response (TIR). To address this problem, an improved algorithm is employed for target detection by maximizing the detection probability of the received echo on the promise of ensuring the TIR estimation precision. In this algorithm, an additional weight vector is introduced to achieve a trade-off among different targets. Both the estimate of TIR and transmit waveform can be updated at each step based on the previous step. Under the same constraint on waveform energy and bandwidth, the information theoretical approach is also considered. In addition, the relationship between the waveforms that are designed based on the two criteria is discussed. Unlike most existing works that only consider single target with temporally correlated characteristics, waveform design for multiple extended targets is considered in this method. Simulation results demonstrate that compared with linear frequency modulated (LFM) signal, waveforms designed based on maximum detection probability and maximum mutual information (MI) criteria can make radar echoes contain more multiple-target information and improve radar performance as a result.

Domain adaptation of lattice-free MMI based TDNN models for speech recognition

The recent proposed time-delay deep neural network (TDNN) acoustic models trained with lattice-free maximum mutual information (LF-MMI) criterion have been shown to give significant performance improvements over other deep neural network (DNN) models in variety speech recognition tasks. Meanwhile, the Kullback---Leibler divergence (KLD) regularization has been validated as an effective adaptation method for DNN acoustic models. However, to our best knowledge, no work has been reported on investigating whether the KLD-based method is also effective for LF-MMI based TDNN models, especially for the domain adaptation. In this study, we generalized the KLD regularized model adaptation to train domain-specific TDNN acoustic models. A few distinct and important observations have been obtained. Experiments were performed on the Cantonese accent, in-car and far-field noise Mandarin speech recognition tasks. Results demonstrated that the proposed domain adapted models can achieve around relative 7---29% word error rate reduction on these tasks, even when the adaptation utterances are only around 1 K.

Analysis of LFM-waveform Libraries for Cognitive Tracking Maneuvering Targets

Based on the idea of the waveform agility in cognitive radars，the waveform libraries for maneuvering target tracking are discussed. LFM-waveform libraries are designed according to different combinations of chirp parameters and FrFT rotation angles. By applying the interact multiple model (IMM) algorithm in tracking maneuvering targets, transmitted waveform is called real time from the LFM-waveform libraries. The waveforms are selected from the library according to the criterion of maximum mutual information between the current state of knowledge of the model and the measurement. Simulation results show that waveform library containing certain amount LFM-waveforms can improve the performance of cognitive tracking radar.

Unsupervised feature selection based on maximum information and minimum redundancy for hyperspectral images

Unsupervised feature selection plays an important role in hyperspectral image processing. It is a very challenge issue to select an effective feature subset with the unavailability of class labels. To select the features maximally preserving the information of original features, a maximum joint mutual information (MJMI) criterion is defined. Since the high-order distribution involved in MJMI is hard to calculate, a maximum information and minimum redundancy (MIMR) criterion is derived as the low-order approximation of MJMI. From information theory, many classical unsupervised feature selection criteria can also be considered as the low-order approximations of MJMI. Compared with them, MIMR requires more relaxed approximation condition. Moreover, a new clonal selection algorithm (CSA) in artificial immune system is devised to optimize the selected features with the guidance of MIMR. Experimental results on several hyperspectral datasets demonstrate that the proposed method obtains better feature subsets compared with classical unsupervised feature selection methods.

Differenced maximum mutual information criterion for robust unsupervised acoustic model adaptation

Discriminative criteria have been widely used for training acoustic models for automatic speech recognition (ASR). Many discriminative criteria have been proposed including maximum mutual information (MMI), minimum phone error (MPE), and boosted MMI (BMMI). Discriminative training is known to provide significant performance gains over conventional maximum-likelihood (ML) training. However, as discriminative criteria aim at direct minimization of the classification error, they strongly rely on having accurate reference labels. Errors in the reference labels directly affect the performance. Recently, the differenced MMI (dMMI) criterion has been proposed for generalizing conventional criteria such as BMMI and MPE. dMMI can approach BMMI or MPE if its hyper-parameters are properly set. Moreover, dMMI introduces intermediate criteria that can be interpreted as smoothed versions of BMMI or MPE. These smoothed criteria are robust to errors in the reference labels. In this paper, we demonstrate the effect of dMMI on unsupervised speaker adaptation where the reference labels are estimated from a first recognition pass and thus inevitably contain errors. In particular, we introduce dMMI-based linear regression (dMMI-LR) adaptation and demonstrate significant gains in performance compared with MLLR and BMMI-LR in two large vocabulary lecture recognition tasks.

Gaussian Mixture Models Reduction by Variational Maximum Mutual Information

Gaussian mixture models (GMMs) are widely used in a variety of classification tasks where it is often important to approximate high order models by models with fewer components. The paper proposes a novel approach to this problem based on a parametric realization of the maximum mutual information (MMI) criterion and its approximation by a closed-form expression named variational-MMI (VMMI). The maximization of the VMMI can be carried out in an analytically tractable manner and it aims at improving the discrimination ability of the reduced set of models, a goal that was not targeted in previous approaches that simplify each class-related GMM independently. Two effective algorithms are proposed and studied for the optimization of the VMMI criterion. One is a steepest descent type algorithm, and the other, called line search A-functions (LSAF), uses concave associated functions. Experiments held in two speech related tasks, phone recognition and language recognition, demonstrate that the VMMI-based parametric model reduction algorithms significantly outperform previous non-discriminative methods. According to these experiments, the EM-like LSAF-based algorithm requires less iterations and converges to a better value of the objective function compared to the steepest descent algorithm.

Source cell-phone recognition from recorded speech using non-speech segments

In a recent study, we have introduced the problem of identifying cell-phones using recorded speech and shown that speech signals convey information about the source device, making it possible to identify the source with some accuracy. In this paper, we consider recognizing source cell-phone microphones using non-speech segments of recorded speech. Taking an information-theoretic approach, we use Gaussian Mixture Model (GMM) trained with maximum mutual information (MMI) to represent device-specific features. Experimental results using Mel-frequency and linear frequency cepstral coefficients (MFCC and LFCC) show that features extracted from the non-speech segments of speech contain higher mutual information and yield higher recognition rates than those from speech portions or the whole utterance. Identification rate improves from 96.42% to 98.39% and equal error rate (EER) reduces from 1.20% to 0.47% when non-speech parts are used to extract features. Recognition results are provided with classical GMM trained both with maximum likelihood (ML) and maximum mutual information (MMI) criteria, as well as support vector machines (SVMs). Identification under additive noise case is also considered and it is shown that identification rates reduces dramatically in case of additive noise.

Joint Access Point Selection and Local Discriminant Embedding for Energy Efficient and Accurate Wi-Fi Positioning

We propose a novel method for improving Wi-Fi positioning accuracy while reducing the energy consumption of mobile devices. Our method presents three contributions. First, we jointly and intelligently select the optimal subset of access points for positioning via maximum mutual information criterion. Second, we further propose local discriminant embedding algorithm for nonlinear discriminative feature extraction, a process that cannot be effectively handled by existing linear techniques. Third, to reduce complexity and make input signal space more compact, we incorporate clustering analysis to localize the positioning model. Experiments in realistic environments demonstrate that the proposed method can lower energy consumption while achieving higher accuracy compared with previous methods. The improvement can be attributed to the capability of our method to extract the most discriminative features for positioning as well as require smaller computation cost and shorter sensing time.

Confidence- and margin-based MMI/MPE discriminative training for off-line handwriting recognition

We present a novel confidence- and margin-based discriminative training approach for model adaptation of a hidden Markov model (HMM)-based handwriting recognition system to handle different handwriting styles and their variations. Most current approaches are maximum-likelihood (ML) trained HMM systems and try to adapt their models to different writing styles using writer adaptive training, unsupervised clustering, or additional writer-specific data. Here, discriminative training based on the maximum mutual information (MMI) and minimum phone error (MPE) criteria are used to train writer-independent handwriting models. For model adaptation during decoding, an unsupervised confidence-based discriminative training on a word and frame level within a two-pass decoding process is proposed. The proposed methods are evaluated for closed-vocabulary isolated handwritten word recognition on the IFN/ENIT Arabic handwriting database, where the word error rate is decreased by 33% relative compared to a ML trained baseline system. On the large-vocabulary line recognition task of the IAM English handwriting database, the word error rate is decreased by 25% relative.

Stochastic gradient identification of Wiener system with maximum mutual information criterion

This study presents an information-theoretic approach for adaptive identification of an unknown Wiener system. A two-criterion identification scheme is proposed, in which the adaptive system comprises a linear finite-impulse response filter trained by maximum mutual information (MaxMI) criterion and a polynomial non-linearity learned by traditional mean square error criterion. The authors show that under certain conditions, the optimum solution matches the true system exactly. Further, the authors develop a stochastic gradient-based algorithm, that is, stochastic mutual information gradient–normalised least mean square algorithm, to implement the proposed identification scheme. Monte-Carlo simulation results demonstrate the noticeable performance improvement of this new algorithm in comparison with some other algorithms.

Performance evaluation of multiple classification of the ultrasonic supraspinatus images by using ML, RBFNN and SVM classifiers

This paper proposed an effort to apply the several multi-class classifiers that are the maximum likelihood classifier, the radial basis function neural network, the fuzzy support vector machine and the error correcting output codes method to classify the ultrasonic supraspinatus images. The maximum mutual information criterion is adopted to search for the powerful features generating from the first order histogram statistics, gray-level co-occurrence matrix and texture feature coding method. In experiments, the most commonly used performance measures including the accuracy, sensitivity, accuracy and F_score are applied to evaluate the classification of the four classifiers. In addition, the Youden’s index, the discriminant power and the area of receiver operating characteristics curve are also used to analyze the classification capability. The experimental results demonstrate that the implementation of radial bass function neural network can achieve 94.1% classification accuracy and performance measures are significantly superior to the others.

SNR-Adaptive Stream Weighting for Audio-MES ASR

Myoelectric signals (MESs) from the speaker's mouth region have been successfully shown to improve the noise robustness of automatic speech recognizers (ASRs), thus promising to extend their usability in implementing noise-robust ASR. In the recognition system presented herein, extracted audio and facial MES features were integrated by a decision fusion method, where the likelihood score of the audio-MES observation vector was given by a linear combination of class-conditional observation log-likelihoods of two classifiers, using appropriate weights. We developed a weighting process adaptive to SNRs. The main objective of the paper involves determining the optimal SNR classification boundaries and constructing a set of optimum stream weights for each SNR class. These two parameters were determined by a method based on a maximum mutual information criterion. Acoustic and facial MES data were collected from five subjects, using a 60-word vocabulary. Four types of acoustic noise including babble, car, aircraft, and white noise were acoustically added to clean speech signals with SNR ranging from -14 to 31 dB. The classification accuracy of the audio ASR was as low as 25.5%. Whereas, the classification accuracy of the MES ASR was 85.2%. The classification accuracy could be further improved by employing the proposed audio-MES weighting method, which was as high as 89.4% in the case of babble noise. A similar result was also found for the other types of noise.

Audio–Visual Affective Expression Recognition Through Multistream Fused HMM

Advances in computer processing power and emerging algorithms are allowing new ways of envisioning human-computer interaction. Although the benefit of audio-visual fusion is expected for affect recognition from the psychological and engineering perspectives, most of existing approaches to automatic human affect analysis are unimodal: information processed by computer system is limited to either face images or the speech signals. This paper focuses on the development of a computing algorithm that uses both audio and visual sensors to detect and track a user's affective state to aid computer decision making. Using our multistream fused hidden Markov model (MFHMM), we analyzed coupled audio and visual streams to detect four cognitive states (interest, boredom, frustration and puzzlement) and seven prototypical emotions (neural, happiness, sadness, anger, disgust, fear and surprise). The MFHMM allows the building of an optimal connection among multiple streams according to the maximum entropy principle and the maximum mutual information criterion. Person-independent experimental results from 20 subjects in 660 sequences show that the MFHMM approach outperforms face-only HMM, pitch-only HMM, energy-only HMM, and independent HMM fusion, under clean and varying audio channel noise condition.

Adaptive filtering under maximum mutual information criterion

The maximum mutual information (MaxMI) criterion is used as the adaptation cost for the adaptive filtering. This criterion is robust to measure distortions, and has strong connection with traditional mean-square error (MSE) criterion. Under Gaussian assumption, the closed-form solution of the finite impulse response (FIR) filter is obtained. Further, based on the kernel density estimation, the stochastic mutual information gradient (SMIG) algorithm is derived. Simulation results emphasize the robustness of this new algorithm.

Training-Based MIMO Systems—Part I: Performance Comparison

In this paper, we make a performance comparison between two different training-based schemes for multiple-input multiple-output (MIMO) channel estimation. The two schemes are the conventional time-multiplexed pilot (CP) scheme and the more recently suggested superimposed pilot (SIP) scheme. Unlike previous comparisons found in the literature, which are mostly based on estimation error performances, the performance comparison in this paper is made by deriving and comparing the maximum data rate (or rather a tight lower bound on the maximum mutual information) achieved by each scheme. By using the maximum mutual information criterion, for each training scheme, we can optimally allocate the time and power spent on transmission of training and data sequences. Once the system parameters (time and power) are tuned to give optimal performance, we can compare their respective maximum data rate. The theory is applied to a blockwise flat-fading MIMO channel, and it is found that in certain scenarios (such as many receive antennas and/or short channel coherence times), it is beneficial to use the SIP. In other scenarios, the SIP scheme suffers from a higher estimation error, and its gain over the CP scheme is often lost.

Linear Spectral Transformation for Robust Speech Recognition Using Maximum Mutual Information

This paper presents a transformation-based rapid adaptation technique for robust speech recognition using a linear spectral transformation (LST) and a maximum mutual information (MMI) criterion. Previously, a maximum likelihood linear spectral transformation (ML-LST) algorithm was proposed for fast adaptation in unknown environments. Since the MMI estimation method does not require evenly distributed training data and increases the a posteriori probability of the word sequences of the training data, we combine the linear spectral transformation method and the MMI estimation technique in order to achieve extremely rapid adaptation using only one word of adaptation data. The proposed algorithm, called MMI-LST, was implemented using the extended Baum-Welch algorithm and phonetic lattices, and evaluated on the TIMIT and FFMTIMIT corpora. It provides a relative reduction in the speech recognition error rate of 11.1% using only 0.25 s of adaptation data.

Discriminative speaker adaptation using articulatory features

This paper presents an automatic speech recognition system using acoustic models based on both sub-phonetic units and broad, phonological features such as voiced and round as output densities in a hidden Markov model framework. The aim of this work is to improve speech recognition performance particularly on conversational speech by using units other than phones as a basis for discrimination between words. We explore the idea that phones are more of a short-hand notation for a bundle of phonological features, which can also be used directly to distinguish competing word hypotheses. Acoustic models for different features are integrated with phone models using a multi-stream approach and log-linear interpolation. This paper presents a new lattice based discriminative training algorithm using the maximum mutual information criterion to train stream weights. This algorithm allows us to automatically learn stream weights from training or adaptation data and can also be applied to other tasks. Decoding experiments conducted in comparison to a non-feature baseline system on the large vocabulary English Spontaneous Scheduling Task show reductions in word error rate of about 20% for discriminative model adaptation based on articulatory features, slightly outperforming other adaptation algorithms.