Maximum Entropy Discrimination Research Articles

Twin maximum entropy discriminations for classification

Maximum entropy discrimination (MED) is an excellent classification method based on the maximum entropy and maximum margin principles, and can produce hard-margin support vector machines (SVMs) under certain condition. In this paper, we propose a novel maximum entropy discrimination classifier called twin maximum entropy discriminations (TMED) which construct two discrimination functions for two classes such that each discrimination function is closer to one of the two classes and is at least γt distance from the other. Therefore, it is more flexible and has better generalization ability than typical MED. Furthermore, it solves a pair of convex optimization problems and has the same advantages as those of non-parallel SVM (NPSVM) which is only the special case of our TMED when the priors and parameters are chosen appropriately. It also owns the inherent sparseness as MED. Experimental results confirm the effectiveness of our proposed method.

Four-Sequence Maximum Entropy Discrimination Algorithm for Glioma Grading

Grading of glioma is crucial for treatment decision making as well as prognostic assessments. In clinical routines, radiologists grade gliomas with multiple complementary magnetic resonance imaging (MRI) sequences, which is yet challenging for glioma prediction models. In this paper, we take full advantages of four commonly used MRI sequences to propose non-invasive grading of glioma based on a variant of maximum entropy discrimination (MED) and decision tree. First, radiomics features calculation is, respectively, performed on T1-weighted imaging, T2-weighted imaging, fluid attenuation inversion recovery imaging, and contrast-enhanced T1-weighted imaging. Then, radiomics features are integrated to build a glioma prediction model named four-sequence MED (FSMED) according to the assumption that the classification margin of different sequences is consistent. Finally, we propose a multi-MED decision tree (MMEDT) model to obtain the grading of gliomas based on the output of FSMED and the results of MED on each sequence. Validation experiments are conducted on a data set collected from Henan Provincial People's Hospital (GliomaHPPH2018) and Multimodal Brain Tumor Image Segmentation Benchmark 2017 (BraTS2017). The results of these two data sets demonstrate the high-prediction performance of our method. The average areas under the curve (AUC) of MMEDT are 0.9119, 0.8184, and 0.9084 for GliomaHPPH2018, BraTS2017, and their merged set, respectively, with the corresponding average sensitivities of 92.55%, 87.85%, and 87.91%, and average specificities of 92.57%, 81.36%, and 87.39%.

General multi-view learning with maximum entropy discrimination

Maximum entropy discrimination (MED) is an excellent discriminative classification algorithm in view of maximum entropy and maximum margin principles, and can generate hard-margin support vector machines (SVMs) under certain condition. Recently, many versions of multi-view maximum entropy discrimination are proved to be effective in multi-view learning. However, those methods are designed only for the two-view classification problem. For example, alternative multi-view maximum entropy discrimination (AMvMED) and multi-kernel multi-view maximum entropy discrimination (MkMvMED) belong to the two-view classification method. In this paper, we extend AMvMED and MkMvMED to the general multi-view classification problems by jointly learning multiple different views in a non-pairwise way called general alternative multi-view maximum entropy discrimination (GAMvMED) and general multi-kernel multi-view maximum entropy discrimination (GM3ED). In GAMvMED, combination weight for the reconstruction of each view in regularization terms is learned to explore complementarity information among different views. Finally, we provide an efficient iteration algorithm with the classical convex quadratic programming for optimization. Simultaneously, we derive that GAMvMED and GM3ED are equivalent in form under the similar prior assumption. Experimental results confirm the effectiveness of our proposed methods.

Multimodal correlation deep belief networks for multi-view classification

The Restricted Boltzmann machine (RBM) has been proven to be a powerful tool in many specific applications, such as representational learning, document modeling, and many other learning tasks. However, the extensions of the RBM are rarely used in the field of multi-view learning. In this paper, we present a new RBM model based on canonical correlation analysis, named as the correlation RBM, for multi-view learning. The correlation RBM computes multiple representations by regularizing the marginal likelihood function with the consistency among representations from different views. In addition, the multimodal deep model can obtain a unified representation that fuses multiple representations together. Therefore, we stack the correlation RBM to create the correlation deep belief network (DBN), and then propose the multimodal correlation DBN for learning multi-view data representations. Contrasting with existing multi-view classification methods, such as multi-view Gaussian process with posterior consistency (MvGP) and consensus and complementarity based maximum entropy discrimination (MED-2C), the correlation RBM and the multimodal correlation DBN have achieved satisfactory results on two-class and multi-class classification datasets. Experimental results show that correlation RBM and the multimodal correlation DBN are effective learning algorithms.

Multi-view learning for visual violence recognition with maximum entropy discrimination and deep features

Harmful information full of violence, whose transmission is difficult to control, is one of the negative effects that the Internet brings to us today. Such corrosive information in the network environment infiltrates the mental health of adolescents. Many young people who lack judgment tend to imitate violent behaviors that are propagated on the network. This study applies multi-view learning to violent behavior recognition of still images, and can be useful for the research of network image or video information monitoring and filtering. In this paper, to start with, we construct a violence image recognition database, which contains 5974 violence images and 11,516 non-violence images. From this database, different datasets can be constructed for model and algorithm evaluations. Then we propose a multi-view maximum entropy discriminant (MVMED+) model for learning with different numbers of views. It can combine various features of the images and thus classify the images with the complementary information between the views. For efficient optimization, we further derive a sequential minimal optimization algorithm to train the model. In the experiments, we use handcrafted features and deep learning features to validate the effectiveness of the proposed model, respectively. As a useful byproduct, large performance improvements on visual violence recognition are further observed with deep learning features.

Semi-supervised soft margin consistency based multi-view maximum entropy discrimination

Multi-view maximum entropy discrimination (MVMED) and alternative MVMED (AMVMED) are proposed as extensions of maximum entropy discrimination (MED). In MVMED and AMVMED, they use hard margin consistency principle that the decision of margin parameter is related to classifier parameter directly. While the decision always be indirectly in practice, thus soft margin consistency based multi-view maximum entropy discrimination (SMVMED) has been proposed. But it is found that SMVMED is only adaptive to supervised problems. In this paper, we extend the model of SMVMED to the semi-supervised problems and develop a semi-supervised SMVMED (SSMVMED). Related experiments on multi-view data sets from different aspects have validated the effectiveness of SSMVMED theoretically and empirically. From the experiments, it is found that (1) compared with SMVMED, the average test accuracy of SSMVMED has a 2% enhancement; (2) SSMVMED costs more training time than SMVMED and the extra time is not more than 10%; (3) in terms of the generation of additional unlabeled instances, ‘mid’ strategy has a better test accuracy than ‘self’ and taking all instances to get the center brings a better test accuracy as well; (4) with SSMVMED, the applications to estimation problem and regression problem will be more feasible.

Latent subspace modeling of sequential data under the maximum entropy discrimination framework

Abstract Hidden Markov models (HMMs) are a popular approach for modeling continuous sequential data, typically based on the assumption of Gaussian-distributed observations. A significant issue HMMs with Gaussian conditional densities are confronted with concerns effectively modeling high-dimensional observations, without getting prone to overfitting or singularities. To this end, one can resort to extracting lower-dimensional latent variable representations of the observed high-dimensional data, as part of the inference algorithm of the postulated HMM. Factor analysis (FA) is a well-established linear latent variable scheme that can be employed for this purpose; its functionality consists in modeling the covariances between the elements of multivariate observations under a set of linear assumptions. Recently, it has been proposed that FA can be effectively generalized under an efficient large-margin Bayesian inference perspective, namely maximum entropy discrimination (MED). This work capitalizes on these recent findings to derive an effective HMM-driven sequential data modeling framework for high-dimensional data. Our proposed approach extracts lower-dimensional latent variable representations of observed high-dimensional data, taking into account the large-margin principle. On this basis, it postulates that the data temporal dynamics are conditional to the inferred values of these latent variables. We devise efficient mean-field inference algorithms for our model, and exhibit its advantages through a set of experiments.

Latent Laplacian Maximum Entropy Discrimination for Detection of High-Utility Anomalies

Data-driven anomaly detection methods suffer from the drawback of detecting all instances that are statistically rare, irrespective of whether the detected instances have real-world significance or not. In this paper, we are interested in the problem of specifically detecting anomalous instances that are known to have high real-world utility, while ignoring the low-utility statistically anomalous instances. To this end, we propose a novel method called Latent Laplacian Maximum Entropy Discrimination (LatLapMED) as a potential solution. This method uses the Entropy Minimization (EM) algorithm to simultaneously incorporate the Geometric EM principle for identifying statistical anomalies, and the MED principle to incorporate utility labels, in order to detect high-utility anomalies. We apply our method in both simulated and real datasets to demonstrate that it has superior performance over existing alternatives that independently pre-process with unsupervised anomaly detection algorithms before classifying.

Semi-supervised multi-view maximum entropy discrimination with expectation Laplacian regularization

Semi-supervised multi-view learning has attracted considerable attention and achieved great success in the machine learning field. This paper proposes a semi-supervised multi-view maximum entropy discrimination approach (SMVMED) with expectation Laplacian regularization for data classification. It takes advantage of the geometric information of the marginal distribution embedded in unlabeled data to construct a semi-supervised classifier. Different from existing methods using Laplacian regularization, we propose to use expectation Laplacian regularization for semi-supervised learning in probabilistic models. We give two implementations of SMVMED and provide their kernel variants. One of them can be relaxed and formulated as a quadratic programming problem that is solved easily. Therefore, for this implementation, we provided two versions which are approximate and exact ones. The experiments on one synthetic and multiple real-world data sets show that SMVMED demonstrates superior performance over semi-supervised single-view maximum entropy discrimination, MVMED and other state-of-the-art semi-supervised multi-view learning methods.

Learning to Classify With Possible Sensor Failures

In this paper, we propose a general framework to learn a robust large-margin binary classifier when corrupt measurements, called anomalies, caused by sensor failure might be present in the training set. The goal is to minimize the generalization error of the classifier on noncorrupted measurements while controlling the false alarm rate associated with anomalous samples. By incorporating a nonparametric regularizer based on an empirical entropy estimator, we propose a geometric-entropy-minimization regularized maximum entropy discrimination (GEM-MED) method to learn to classify and detect anomalies in a joint manner. We demonstrate using simulated data and a real multimodal data set. Our GEM-MED method can yield improved performance over previous robust classification methods in terms of both classification accuracy and anomaly detection rate.

Maximum entropy discrimination factor analyzers

Devising generative models that allow for inferring low dimensional latent feature representations of high-dimensional observations is a significant problem in statistical machine learning. Factor analysis (FA) is a well-established linear latent variable scheme addressing this problem by modeling the covariances between the elements of multivariate observations under a set of linear assumptions. FA is closely related to principal components analysis (PCA), and might be considered as a generalization of both PCA and its probabilistic version, PPCA. Recently, the invention of Gaussian process latent variable models (GP-LVMs) has given rise to a whole new family of latent variable modeling schemes that generalize FA under a nonparametric Bayesian inference framework. In this work, we examine generalization of FA models under a different Bayesian inference perspective. Specifically, we propose a large-margin formulation of FA under the maximum entropy discrimination (MED) framework. The MED framework integrates the large-margin principle with Bayesian posterior inference in an elegant and computationally efficient fashion, allowing to leverage existing high-performance solvers for convex optimization problems. We devise efficient mean-field inference algorithms for our model, and exhibit its advantages by evaluating it in a number of diverse application scenarios, dealing with high-dimensional data classification and reconstruction.

Consensus and complementarity based maximum entropy discrimination for multi-view classification

Maximum entropy discrimination (MED) is a general framework for discriminative estimation based on the maximum entropy and large margin principles, but it just uses only one view of the data not all the views of the data. Although Multi-view maximum entropy discrimination considered the multi-view information of the data, it just respects the consensus principle. There are two common principles named consensus and complementarity in multi-view learning. We aim to take the full advantage of the multi-view information of the data for classification, and respect the two common principles consensus and complementarity simultaneously. In this paper, we propose a new method called consensus and complementarity based MED (MED-2C) for multi-view classification, which well utilizes the two principles consensus and complementarity for multi-view learning (MVL). We first transform data from two views into a common subspace and make the transformed data in the new subspace identical to respect the consensus principle. Then we augment the transformed data with their original features to take into account the complementarity principle. Built on the augmented features and meanwhile by relaxing the consensus principle, the objective function of MED-2C is naturally formed whose inner optimization recovers the traditional MED framework. We provide an instantiation of the MED-2C method and derive the corresponding solution. Experimental results on synthetic data and real-world data show the effectiveness of the proposed MED-2C. It not only performs better than three single-view classification methods but also generally outperforms three multi-view classification methods canonical correlation analysis (CCA), ensemble MED (EMED) and the multi-view variant SVM-2K of the classical support vector machine (SVM) algorithm. In addition, MED-2C performs better than or as well as state-of-the-art MVMED on all the data sets.

Multi-kernel maximum entropy discrimination for multi-view learning

Maximum entropy discrimination (MED) is a general framework for discriminative estimation which integrates the principles of maximum entropy and maximum margin. In this paper, we propose a novel approach named multi-kernel MED (MKMED) for multi-view learning (MVL), which takes advantage of the comp lementary principle for MVL. Multiple kernels encode the similarities in different views. We obtain a kernel matrix by multiple kernel combination to make use of the complementary information in different views. Based on the kernel matrix obtained by multiple kernel combination, we can proceed MVL within the MED framework. The experimental results on multiple datasets demonstrate the effectiveness of the proposed MKMED. MKMED outperforms the single-view MEDs and a competing MVL mothod named SVM-2K, and is competitive with the state-of-the-art multi-view MED (MVMED) and even sometimes exceeds it.

Alternative Multiview Maximum Entropy Discrimination.

Maximum entropy discrimination (MED) is a general framework for discriminative estimation based on maximum entropy and maximum margin principles, and can produce hard-margin support vector machines under some assumptions. Recently, the multiview version of MED multiview MED (MVMED) was proposed. In this paper, we try to explore a more natural MVMED framework by assuming two separate distributions p1( Θ1) over the first-view classifier parameter Θ1 and p2( Θ2) over the second-view classifier parameter Θ2 . We name the new MVMED framework as alternative MVMED (AMVMED), which enforces the posteriors of two view margins to be equal. The proposed AMVMED is more flexible than the existing MVMED, because compared with MVMED, which optimizes one relative entropy, AMVMED assigns one relative entropy term to each of the two views, thus incorporating a tradeoff between the two views. We give the detailed solving procedure, which can be divided into two steps. The first step is solving our optimization problem without considering the equal margin posteriors from two views, and then, in the second step, we consider the equal posteriors. Experimental results on multiple real-world data sets verify the effectiveness of the AMVMED, and comparisons with MVMED are also reported.

Multi-level max-margin analysis for semantic classification of satellite images

The performance of scene classification of satellite images strongly relies on the discriminative power of the low-level and mid-level feature representation. This paper presents a novel approach, named multi-level max-margin analysis (M3DA) for semantic classification for high-resolution satellite images. In our M3DA model, the maximum entropy discrimination latent Dirichlet allocation (MedLDA) model is applied to learn the topic-level features first, and then based on a bag-of-words representation of low-level local image features, the large margin nearest neighbor (LMNN) classifier is used to optimize a multiple soft label composed of word-level features (generated by SVM classifier) and topic-level features. The categorization performances on 21-class land-use dataset have demonstrated that the proposed model in multi-level max-margin scheme can distinguish different categories of land-use scenes reasonably.

Object-Oriented Semisupervised Classification of VHR Images by Combining MedLDA and a Bilateral Filter

A Bayesian hierarchical model is presented to classify very high resolution (VHR) images in a semisupervised manner, in which both a maximum entropy discrimination latent Dirichlet allocation (MedLDA) and a bilateral filter are combined into a novel application framework. The primary contribution of this paper is to nullify the disadvantages of traditional probabilistic topic models on pixel-level supervised information and to achieve the effective classification of VHR remote sensing images. This framework consists of the following two iterative steps. In the training stage, the model utilizes the central labeled pixel and its neighborhood, as a squared labeled image object, to train the classifiers. In the classification stage, each central unlabeled pixel with its neighborhood, as an unlabeled object, is classified as a user-provided geoobject class label with the maximum posterior probability. Gibbs sampling is adopted for model inference. The experimental results demonstrate that the proposed method outperforms two classical SVM-based supervised classification methods and probabilistic-topic-models-based classification methods.

Generalized multiple kernel learning with data-dependent priors.

Multiple kernel learning (MKL) and classifier ensemble are two mainstream methods for solving learning problems in which some sets of features/views are more informative than others, or the features/views within a given set are inconsistent. In this paper, we first present a novel probabilistic interpretation of MKL such that maximum entropy discrimination with a noninformative prior over multiple views is equivalent to the formulation of MKL. Instead of using the noninformative prior, we introduce a novel data-dependent prior based on an ensemble of kernel predictors, which enhances the prediction performance of MKL by leveraging the merits of the classifier ensemble. With the proposed probabilistic framework of MKL, we propose a hierarchical Bayesian model to learn the proposed data-dependent prior and classification model simultaneously. The resultant problem is convex and other information (e.g., instances with either missing views or missing labels) can be seamlessly incorporated into the data-dependent priors. Furthermore, a variety of existing MKL models can be recovered under the proposed MKL framework and can be readily extended to incorporate these priors. Extensive experiments demonstrate the benefits of our proposed framework in supervised and semisupervised settings, as well as in tasks with partial correspondence among multiple views.

Tile-Level Annotation of Satellite Images Using Multi-Level Max-Margin Discriminative Random Field

This paper proposes a multi-level max-margin discriminative analysis (M3DA) framework, which takes both coarse and fine semantics into consideration, for the annotation of high-resolution satellite images. In order to generate more discriminative topic-level features, the M3DA uses the maximum entropy discrimination latent Dirichlet Allocation (MedLDA) model. Moreover, for improving the spatial coherence of visual words neglected by M3DA, conditional random field (CRF) is employed to optimize the soft label field composed of multiple label posteriors. The framework of M3DA enables one to combine word-level features (generated by support vector machines) and topic-level features (generated by MedLDA) via the bag-of-words representation. The experimental results on high-resolution satellite images have demonstrated that, using the proposed method can not only obtain suitable semantic interpretation, but also improve the annotation performance by taking into account the multi-level semantics and the contextual information.

Maximum Entropy Discrimination Markov Networks

The standard maximum margin approach for structured prediction lacks a straightforward probabilistic interpretation of the learning scheme and the prediction rule. Therefore its unique advantages such as dual sparseness and kernel tricks cannot be easily conjoined with the merits of a probabilistic model such as Bayesian regularization, model averaging, and ability to model hidden variables. In this paper, we present a new general framework called maximum entropy discrimination Markov networks (MaxEnDNet, or simply, MEDN), which integrates these two approaches and combines and extends their merits. Major innovations of this approach include: 1) It extends the conventional max-entropy discrimination learning of classification rules to a new structural max-entropy discrimination paradigm of learning a distribution of Markov networks. 2) It generalizes the extant Markov network structured-prediction rule based on a point estimator of model coefficients to an averaging model akin to a Bayesian predictor that integrates over a learned posterior distribution of model coefficients. 3) It admits flexible entropic regularization of the model during learning. By plugging in different prior distributions of the model coefficients, it subsumes the well-known maximum margin Markov networks (M3N) as a special case, and leads to a model similar to an L1-regularized M3N that is simultaneously primal and dual sparse, or other new types of Markov networks. 4) It applies a modular learning algorithm that combines existing variational inference techniques and convex-optimization based M3N solvers as subroutines. Essentially, MEDN can be understood as a jointly maximum likelihood and maximum margin estimate of Markov network. It represents the first successful attempt to combine maximum entropy learning (a dual form of maximum likelihood learning) with maximum margin learning of Markov network for structured input/output problems; and the basic principle can be generalized to learning arbitrary graphical models, such as the generative Bayesian networks or models with structured hidden variables. We discuss a number of theoretical properties of this approach, and show that empirically it outperforms a wide array of competing methods for structured input/output learning on both synthetic and real OCR and web data extraction data sets.

Joint acoustic and language modeling for speech recognition

In a traditional model of speech recognition, acoustic and linguistic information sources are assumed independent of each other. Parameters of hidden Markov model (HMM) and n-gram are separately estimated for maximum a posteriori classification. However, the speech features and lexical words are inherently correlated in natural language. Lacking combination of these models leads to some inefficiencies. This paper reports on the joint acoustic and linguistic modeling for speech recognition by using the acoustic evidence in estimation of the linguistic model parameters, and vice versa, according to the maximum entropy (ME) principle. The discriminative ME (DME) models are exploited by using features from competing sentences. Moreover, a mutual ME (MME) model is built for sentence posterior probability, which is maximized to estimate the model parameters by characterizing the dependence between acoustic and linguistic features. The N-best Viterbi approximation is presented in implementing DME and MME models. Additionally, the new models are incorporated with the high-order feature statistics and word regularities. In the experiments, the proposed methods increase the sentence posterior probability or model separation. Recognition errors are significantly reduced in comparison with separate HMM and n-gram model estimations from 32.2% to 27.4% using the MATBN corpus and from 5.4% to 4.8% using the WSJ corpus (5K condition).