Minimax Entropy Research Articles

Dual Classifier Adaptation: Source-Free UDA via Adaptive Pseudo-Labels Learning

Different from Unsupervised Domain Adaptation (UDA), Source-Free Unsupervised Domain Adaptation (SFUDA) transfers source knowledge to target domain without accessing the source data, using only the source model, has attracted much attention recently. One mainstream SFUDA method fine-tunes the source model by self-training to generate pseudo-labels of the target data. However, due to the significant differences between different domains, these target pseudo-labels often contain some noise, and it will inevitably degenerates the target performance. For this purpose, we propose an innovative SFUDA method with adaptive pseudo-labels learning named Dual Classifier Adaptation (DCA). In DCA, a dual classifier structure is introduced to adaptively learn target pseudo-labels by cooperation between source and target classifiers. Simultaneously, the minimax entropy is introduced for target learning, in order to adapt target data to source model, while capture the intrinsic cluster structure in target domain as well. After compared our proposed method DCA with a range of UDA and SFUDA methods, DCA achieves far ahead performance on several benchmark datasets.

Human-level few-shot concept induction through minimax entropy learning.

Humans learn concepts both from labeled supervision and by unsupervised observation of patterns, a process machines are being taught to mimic by training on large annotated datasets-a method quite different from the human pathway, wherein few examples with no supervision suffice to induce an unfamiliar relational concept. We introduce a computational model designed to emulate human inductive reasoning on abstract reasoning tasks, such as those in IQ tests, using a minimax entropy approach. This method combines identifying the most effective constraints on data via minimum entropy with determining the best combination of them via maximum entropy. Our model, which applies this unsupervised technique, induces concepts from just one instance, reaching human-level performance on tasks of Raven's Progressive Matrices (RPM), Machine Number Sense (MNS), and Odd-One-Out (O3). These results demonstrate the potential of minimax entropy learning for enabling machines to learn relational concepts efficiently with minimal input.

Multicentric intelligent cardiotocography signal interpretation using deep semi-supervised domain adaptation via minimax entropy and domain invariance

Background and ObjectiveObstetricians use Cardiotocography (CTG), which is the continuous recording of fetal heart rate and uterine contraction, to assess fetal health status. Deep learning models for intelligent fetal monitoring trained on extensively labeled and identically distributed CTG records have achieved excellent performance. However, creation of these training sets requires excessive time and specialist labor for the collection and annotation of CTG signals. Previous research has demonstrated that multicenter studies can improve model performance. However, models trained on cross-domain data may not generalize well to target domains due to variance in distribution among datasets. Hence, this paper conducted a multicenter study with Deep Semi-Supervised Domain Adaptation (DSSDA) for intelligent interpretation of antenatal CTG signals. This approach helps to align cross-domain distribution and transfer knowledge from a label-rich source domain to a label-scarce target domain. MethodsWe proposed a DSSDA framework that integrated Minimax Entropy and Domain Invariance (DSSDA-MMEDI) to reduce inter-domain gaps and thus achieve domain invariance. The networks were developed using GoogLeNet to extract features from CTG signals, with fully connected, softmax layers for classification. We designed a Dynamic Gradient-driven strategy based on Mutual Information (DGMI) to unify the losses from Minimax Entropy (MME), Domain Invariance (DI), and supervised cross-entropy during iterative learning. ResultsWe validated our DSSDA model on two datasets collected from collaborating healthcare institutions and mobile terminals as the source and target domains, which contained 16,355 and 3,351 CTG signals, respectively. Compared to the results achieved with deep learning networks without DSSDA, DSSDA-MMEDI significantly improved sensitivity and F1-score by over 6%. DSSDA-MMEDI also outperformed other state-of-the-art DSSDA approaches for CTG signal interpretation. Ablation studies were performed to determine the unique contribution of each component in our DSSDA mechanism. ConclusionsThe proposed DSSDA-MMEDI is feasible and effective for alignment of cross-domain data and automated interpretation of multicentric antenatal CTG signals with minimal annotation cost.

Semi-supervised domain adaptation on graphs with contrastive learning and minimax entropy

Label scarcity in a graph is frequently encountered in real-world applications due to the high cost of data labeling. To this end, semi-supervised domain adaptation (SSDA) on graphs aims to leverage the knowledge of a labeled source graph to aid in node classification on a target graph with limited labels. SSDA tasks need to overcome the domain gap between the source and target graphs. However, to date, this challenging research problem has yet to be formally considered by the existing approaches designed for cross-graph node classification. This paper proposes a novel method called SemiGCL to tackle the graph Semi-supervised domain adaptation with Graph Contrastive Learning and minimax entropy training. SemiGCL generates informative node representations by contrasting the representations learned from a graph’s local and global views. Additionally, SemiGCL is adversarially optimized with the entropy loss of unlabeled target nodes to reduce domain divergence. Experimental results on benchmark datasets demonstrate that SemiGCL outperforms the state-of-the-art baselines on the SSDA tasks.

Fault classification of the rolling bearing based on minimax entropy domain adaption augmented with signal generation algorithm

Rolling bearing faults have been capturing substantial research attention as they are the root causes of malfunctions in mechatronics systems than any other factors. The detection of rolling bearing faults in the early stage is therefore a mandatory requirement demanded by reliable industrial plants. To release the dependence of diagnostic methods on human expertise and system’s understanding, this work proposes a fault classification method for rolling bearings that is based on a deep learning framework. The framework consists of a minimax entropy domain adaptation algorithm augmented with a signal generalization algorithm. The function of the signal generalization algorithm is to reduce the domain shift between training and testing datasets that are often obtained experimentally from different working conditions. The generalized signal is then represented in the form of Fourier series whose coefficients contain intrinsic information that associated with different types of bearing faults. A convolutional neural network extracts the hidden information of bearing faults buried in the Fourier coefficients and then categorises the working condition of the bearing under test. By combining the advantages of both signal processing techniques in the frequency domain and the minimax entropy domain adaptation, the novel diagnostic framework is able to detect bearing faults from different working conditions. The effectiveness of the proposed diagnostic algorithm is experimentally verified by two case studies that were prepared with different types and levels of bearing faults.

Self-supervised learning minimax entropy domain adaptation for the underwater target recognition

With wide research of intelligent methods, studies on underwater target recognition have been making rapid progress. However, various marine conditions may cause data distribution mismatch between the collected signal sets, reducing model recognition performance. To mitigate the negative impact of data divergence, this paper uses the domain adaptation methods in target recognition and proposes an improved domain adaptation frame, self-supervised learning minimax entropy. Firstly, based on the minimax entropy method (MME), the prediction consistency is utilized to determine pseudo-labels, and the loss weight is introduced to deal with the misaligned target domain data. Then, a self-supervised learning mechanism is designed to ensure consistency of prediction results during training. Three different features, including the constant-Q transform (CQT), Mel spectrum, and Mel-frequency cepstral coefficient (MFCC), are used to verify the performance of domain adaptation methods. The experimental results show that applying domain adaptations can effectively improve the recognition performance of the models under most experimental conditions, and the improved frame has higher average recognition accuracy than other domain adaptation methods in the experiments.

Bridging symbolic and subsymbolic reasoning with minimax entropy models

In this paper, we investigate MiniMax Entropy models, a class of neural symbolic models where symbolic and subsymbolic features are seamlessly integrated. We show how these models recover classical algorithms from both the deep learning and statistical relational learning scenarios. Novel hybrid settings are defined and experimentally explored, showing state-of-the-art performance in collective classification, knowledge base completion and graph (molecular) data generation.

SSDAN: Multi-Source Semi-Supervised Domain Adaptation Network for Remote Sensing Scene Classification

We present a new method for multi-source semi-supervised domain adaptation in remote sensing scene classification. The method consists of a pre-trained convolutional neural network (CNN) model, namely EfficientNet-B3, for the extraction of highly discriminative features, followed by a classification module that learns feature prototypes for each class. Then, the classification module computes a cosine distance between feature vectors of target data samples and the feature prototypes. Finally, the proposed method ends with a Softmax activation function that converts the distances into class probabilities. The feature prototypes are also divided by a temperature parameter to normalize and control the classification module. The whole model is trained on both the unlabeled and labeled target samples. It is trained to predict the correct classes utilizing the standard cross-entropy loss computed over the labeled source and target samples. At the same time, the model is trained to learn domain invariant features using another loss function based on entropy computed over the unlabeled target samples. Unlike the standard cross-entropy loss, the new entropy loss function is computed on the model’s predicted probabilities and does not need the true labels. This entropy loss, called minimax loss, needs to be maximized with respect to the classification module to learn features that are domain-invariant (hence removing the data shift), and at the same time, it should be minimized with respect to the CNN feature extractor to learn discriminative features that are clustered around the class prototypes (in other words reducing intra-class variance). To accomplish these maximization and minimization processes at the same time, we use an adversarial training approach, where we alternate between the two processes. The model combines the standard cross-entropy loss and the new minimax entropy loss and optimizes them jointly. The proposed method is tested on four RS scene datasets, namely UC Merced, AID, RESISC45, and PatternNet, using two-source and three-source domain adaptation scenarios. The experimental results demonstrate the strong capability of the proposed method to achieve impressive performance despite using only a few (six in our case) labeled target samples per class. Its performance is already better than several state-of-the-art methods, including RevGrad, ADDA, Siamese-GAN, and MSCN.

A novel multi-adversarial cross-domain neural network for bearing fault diagnosis

Recently, deep neural networks have achieved great success in bearing fault diagnosis. Most existing methods are developed under the assumption that the bearing vibration signals are collected under the same machine operating conditions. However, bearing fault diagnosis under cross-domain conditions will suffer from domain shift problems if the neural network is only trained with the source domain data. Moreover, acquiring enough labeled data from the target domain will be expensive and time-consuming. To address the above problems, this paper proposes an end-to-end multi-adversarial cross-domain neural network for bearing fault diagnosis, which takes labeled source domain data and unlabeled target domain data to achieve the cross-domain bearing fault diagnosis under cross-load conditions and cross-machine conditions. The proposed method employs multi-adversarial training to automatically extract the domain-invariant features from source and target domains instead of manually designing features, which combines domain-adversarial learning and mini-max entropy adversarial learning to adversarially reduce the domain discrepancy between the source and target domains and alleviate the class misalignment problem. The results of the cross-load and the cross-machine experiments prove the effectiveness of the proposed method, and the proposed method provides a promising tool for cross-domain bearing fault diagnosis.

Experimental Validation of Minimax Entropy Principle in Ultrasound Images

Ultrasound imaging is one of the non-invasive imaging, that diagnoses the disease inside a human body and there are numerous ultrasonic devices being used frequently. Entropy as a well known statistical measure of uncertainty has a considerable impact on the medical images. A procedure for minimizing the entropy with respect to the region of interest is demonstrated. This new approach has shown the experiments using Extracted Region Of Interest Based Sharpened image, called as (EROIS) image based on Minimax entropy principle and various filters. In this turn, the approach also validates the versatility of the entropy concept. Experiments have been performed practically on the real-time ultrasound images collected from ultrasound centers and have shown a significant performance. The present approach has been validated with showing results over ultrasound images of the Human Gallbladder.

Application of evidential reasoning approach and OWA operator weights in road safety evaluation considering the best and worst practice frontiers

Data envelopment analysis (DEA) has recently become relatively popular with road safety experts. Therefore, various decision-making units (DMUs), such as EU countries, have been assessed in terms of road safety performance (RSP). However, the DEA has been criticized because it evaluates DMUs based only on the concept of self-assessment, and, therefore does not provide a unique ranking for DMUs. Therefore, cross efficiency method (CEM) was developed to overcome this shortcoming. Peer-evaluations in addition to self-evaluation have made the CEM to be recognized as an effective method for ranking DMUs. The traditional CEM is based only on the standard CCR (Charnes, Cooper and Rhodes) model, and it evaluates DMUs according to their position relative to the best practice frontier while neglecting the worst practice frontier. However, the DMUs can also be assessed based on their position relative to the worst practice frontier. In this regard, the present study aims to provide a double-frontier CEM for assessing RSP by taking into account the best and worst frontiers simultaneously. For this purpose, the cross efficiency and cross anti-efficiency matrices are generated.Even though a weighted average method (WAM) is most frequently used for cross efficiency aggregation, the decision maker's (DM) preference structure may not be reflected. For this reason, the present study mainly focuses on the evidential reasoning approach (ERA), as a nonlinear aggregation method, rather than the linear WAM. Equal weights are often used for cross efficiency aggregation; consequently, the effect of the DM's subjective judgments in obtaining the overall efficiency is ignored. In this respect, the minimax entropy approach (MEA) and the maximum disparity approach (MMDA) are applied for determining the ordered weighted averaging (OWA) operator weights for cross efficiency aggregation. The weighted cross efficiencies and cross anti-efficiencies are then aggregated using the ERA. Finally, the proposed method, called DF-CEM-ERA, is used to evaluate the RSP of EU countries as well as Serbian police departments (PDs).

A Hierarchical and Contextual Model for Aerial Image Parsing

In this paper we present a hierarchical and contextual model for aerial image understanding. Our model organizes objects (cars, roofs, roads, trees, parking lots) in aerial scenes into hierarchical groups whose appearances and configurations are determined by statistical constraints (e.g. relative position, relative scale, etc.). Our hierarchy is a non-recursive grammar for objects in aerial images comprised of layers of nodes that can each decompose into a number of different configurations. This allows us to generate and recognize a vast number of scenes with relatively few rules. We present a minimax entropy framework for learning the statistical constraints between objects and show that this learned context allows us to rule out unlikely scene configurations and hallucinate undetected objects during inference. A similar algorithm was proposed for texture synthesis (Zhu et al. in Int. J. Comput. Vis. 2:107---126, 1998) but didn't incorporate hierarchical information. We use a range of different bottom-up detectors (AdaBoost, TextonBoost, Compositional Boosting (Freund and Schapire in J. Comput. Syst. Sci. 55, 1997; Shotton et al. in Proceedings of the European Conference on Computer Vision, pp. 1---15, 2006; Wu et al. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1---8, 2007)) to propose locations of objects in new aerial images and employ a cluster sampling algorithm (C4 (Porway and Zhu, 2009)) to choose the subset of detections that best explains the image according to our learned prior model. The C4 algorithm can quickly and efficiently switch between alternate competing sub-solutions, for example whether an image patch is better explained by a parking lot with cars or by a building with vents. We also show that our model can predict the locations of objects our detectors missed. We conclude by presenting parsed aerial images and experimental results showing that our cluster sampling and top-down prediction algorithms use the learned contextual cues from our model to improve detection results over traditional bottom-up detectors alone.

Minimax entropy solutions of ill-posed problems

Convergent methodology for ill-posed problems is typically equivalent to application of an operator dependent on a single parameter derived from the noise level and the data (a regularization parameter or terminal iteration number). In the context of a given problem discretized for purposes of numerical analysis, these methods can be viewed as resulting from imposed prior constraints bearing the same amount of information content. We identify a new convergent method for the treatment of certain multivariate ill-posed problems, which imposes constraints of a much lower information content (i.e., having much lower bias), based on the operator’s dependence on many data-derived parameters. The associated marked performance improvements that are possible are illustrated with solution estimates for a Lyapunov equation structured by an ill-conditioned matrix. The methodology can be understood in terms of a Minimax Entropy Principle, which emerges from the Maximum Entropy Principle in some multivariate settings.

Statistical Principles in Image Modeling

Images of natural scenes contain a rich variety of visual patterns. To learn and recognize these patterns from natural images, it is necessary to construct statistical models for these patterns. In this review article we describe three statistical principles for modeling image patterns: the sparse coding principle, the minimax entropy principle, and the meaningful alignment principle. We explain these three principles and their relationships in the context of modeling images as compositions of Gabor wavelets. These three principles correspond to three regimes of composition patterns of Gabor wavelets, and these three regimes are connected by changes in scale or resolution.

Modeling Visual Patterns by Integrating Descriptive and Generative Methods

This paper presents a class of statistical models that integrate two statistical modeling paradigms in the literature: (I) Descriptive methods, such as Markov random fields and minimax entropy learning (Zhu, S.C., Wu, Y.N., and Mumford, D. 1997. Neural Computation, 9(8)), and (II) Generative methods, such as principal component analysis, independent component analysis (Bell, A.J. and Sejnowski, T.J. 1997. Vision Research, 37:3327-3338), transformed component analysis (Frey, B. and Jojic, N. 1999. ICCV), wavelet coding (Mallat, S. and Zhang, Z. 1993. IEEE Trans. on Signal Processing, 41:3397-3415; Chen, S., Donoho, D., and Saunders, M.A. 1999. Journal on Scientific Computing, 20(1):33-61), and sparse coding (Olshausen, B.A. and Field, D.J. 1996. Nature, 381:607-609; Lewicki, M.S. and Olshausen, B.A. 1999. JOSA, A. 16(7): 1587-1601). In this paper, we demonstrate the integrated framework by constructing a class of hierarchical models for texton patterns (the term "texton" was coined by psychologist Julesz in the early 80s). At the bottom level of the model, we assume that an observed texture image is generated by multiple hidden "texton maps", and textons on each map are translated, scaled, stretched, and oriented versions of a window function, like mini-templates or wavelet bases. The texton maps generate the observed image by occlusion or linear superposition. This bottom level of the model is generative in nature. At the top level of the model, the spatial arrangements of the textons in the texton maps are characterized by minimax entropy principle, which leads to embellished versions of Gibbs point process models (Stoyan, D., Kendall, W.S., and Mecke, J. 1985. Stochastic Geometry and its Applications). The top level of the model is descriptive in nature. We demonstrate the integrated model by a set of experiments.

Learning in Gibbsian fields: how accurate and how fast can it be?

Gibbsian fields or Markov random fields are widely used in Bayesian image analysis, but learning Gibbs models is computationally expensive. The computational complexity is pronounced by the recent minimax entropy (FRAME) models which use large neighborhoods and hundreds of parameters. In this paper, we present a common framework for learning Gibbs models. We identify two key factors that determine the accuracy and speed of learning Gibbs models: The efficiency of likelihood functions and the variance in approximating partition functions using Monte Carlo integration. We propose three new algorithms. In particular, we are interested in a maximum satellite likelihood estimator, which makes use of a set of precomputed Gibbs models called "satellites" to approximate likelihood functions. This algorithm can approximately estimate the minimax entropy model for textures in seconds in a HP workstation. The performances of various learning algorithms are compared in our experiments.

Exploring texture ensembles by efficient Markov chain Monte Carlo-Toward a "trichromacy" theory of texture

Presents a mathematical definition of texture, the Julesz ensemble /spl Omega/(h), which is the set of all images (defined on Z/sup 2/) that share identical statistics h. Then texture modeling is posed as an inverse problem: Given a set of images sampled from an unknown /spl Omega/(h/sub */), we search for the statistics h/sub */ which define the ensemble. A /spl Omega/(h) has an associated probability distribution q(I; h), which is uniform over the images in /spl Omega/(h) and has zero probability outside. The authors previously (1999) showed q(I; h) to be the limit distribution of the FRAME (filter, random field, and minimax entropy) model, as the image lattice /spl Lambda//spl rarr/Z/sup 2/. This conclusion establishes the intrinsic link between the scientific definition of texture on Z/sup 2/ and the mathematical models of texture on finite lattices. It brings two advantages: the practice of texture image synthesis by matching statistics is put on a mathematical foundation; and we need not learn the expensive FRAME model in feature pursuit, model selection and texture synthesis. An efficient Markov chain Monte Carte algorithm is proposed for sampling Julesz ensembles. It generates random texture images by moving along the directions of filter coefficients and, thus, extends the traditional single site Gibbs sampler. We compare four popular statistical measures in the literature, in terms of their descriptive abilities. Our experiments suggest that a small number of bins in marginal histograms are sufficient for capturing a variety of texture patterns.

A Minimax Entropy Registration Framework for Patient Setup Verification in Radiotherapy

In external beam radiotherapy (EBRT), patient setup verification over the entire course of fractionated treatment is necessary for accurate delivery of a specified dose to the tumor. We are working on the development of a minimax entropy registration framework for patient setup verification using dual portal images and the treatment planning 3D CT dataset. In this paper, we present an overview of our registration framework, where an iteratively and automatically estimated segmentation of the portal image is utilized to more accurately and robustly register the portal image to the 3D treatment-planning CT data. In addition, we describe initial testing of this approach. We note that, due to low resolution and low contrast of the portal images, this registration presents a difficult problem. We also note that the registration of the images in our proposed method is guided by the bony structure visible in the portal and the 3D CT images. However, since the prostate can move with respect to the pelvic bone, we propose using ultrasound images to quantify this movement.

Embedding Gestalt laws in Markov random fields

The goal of this paper is to study a mathematical framework of 2D object shape modeling and learning for middle level vision problems, such as image segmentation and perceptual organization. For this purpose, we pursue generic shape models which characterize the most common features of 2D object shapes. In this paper, shape models are learned from observed natural shapes based on a minimax entropy learning theory. The learned shape models are Gibbs distributions defined on Markov random fields (MRFs). The neighborhood structures of these MRFs correspond to Gestalt laws-colinearity, cocircularity, proximity, parallelism, and symmetry. Thus, both contour-based and region-based features are accounted for. Stochastic Markov chain Monte Carlo (MCMC) algorithms are proposed for learning and model verification. Furthermore, this paper provides a quantitative measure for the so-called nonaccidental statistics and, thus, justifies some empirical observations of Gestalt psychology by information theory. Our experiments also demonstrate that global shape properties can arise from interactions of local features.

A minimax entropy registration framework for patient setup verification in radiotherapy.

In external beam radiotherapy (EBRT), patient setup verification over the entire course of fractionated treatment is necessary for accurate delivery of a specified dose to the tumor. We are working on the development of a minimax entropy registration framework for patient setup verification using dual portal images and the treatment planning 3D CT dataset. In this paper, we present an overview of our registration framework, where an iteratively and automatically estimated segmentation of the portal image is utilized to more accurately and robustly register the portal image to the 3D treatment-planning CT data. In addition, we describe initial testing of this approach. We note that, due to low resolution and low contrast of the portal images, this registration presents a difficult problem. We also note that the registration of the images in our proposed method is guided by the bony structure visible in the portal and the 3D CT images. However, since the prostate can move with respect to the pelvic bone, we propose using ultrasound images to quantify this movement.