Maximum Entropy Regularization Research Articles

Self-Supervised Deep Multi-Level Representation Learning Fusion-Based Maximum Entropy Subspace Clustering for Hyperspectral Band Selection

As one of the most important techniques for hyperspectral image dimensionality reduction, band selection has received considerable attention, whereas self-representation subspace clustering-based band selection algorithms have received quite a lot of attention with good effect. However, many of them lack the self-supervision of representations and ignore the multi-level spectral–spatial information of HSI and the connectivity of subspaces. To this end, this paper proposes a novel self-supervised multi-level representation learning fusion-based maximum entropy subspace clustering (MLRLFMESC) method for hyperspectral band selection. Firstly, to learn multi-level spectral–spatial information, self-representation subspace clustering is embedded between the encoder layers of the deep-stacked convolutional autoencoder and its corresponding decoder layers, respectively, as multiple fully connected layers to achieve multi-level representation learning (MLRL). A new auxiliary task is constructed for multi-level representation learning and multi-level self-supervised training to improve its capability of representation. Then, a fusion model is designed to fuse the multi-level spectral–spatial information to obtain a more distinctive coefficient matrix for self-expression, where the maximum entropy regularization (MER) method is employed to promote connectivity and the uniform dense distribution of band elements in each subspace. Finally, subspace clustering is conducted to obtain the final band subset. Experiments have been conducted on three hyperspectral datasets, and the corresponding results show that the proposed MLRLFMESC algorithm significantly outperforms several other band selection methods in classification performance.

UNIT: A unified metric learning framework based on maximum entropy regularization

UNIT: A unified metric learning framework based on maximum entropy regularization

Handling Missing Data via Max-Entropy Regularized Graph Autoencoder

Graph neural networks (GNNs) are popular weapons for modeling relational data. Existing GNNs are not specified for attribute-incomplete graphs, making missing attribute imputation a burning issue. Until recently, many works notice that GNNs are coupled with spectral concentration, which means the spectrum obtained by GNNs concentrates on a local part in spectral domain, e.g., low-frequency due to oversmoothing issue. As a consequence, GNNs may be seriously flawed for reconstructing graph attributes as graph spectral concentration tends to cause a low imputation precision. In this work, we present a regularized graph autoencoder for graph attribute imputation, named MEGAE, which aims at mitigating spectral concentration problem by maximizing the graph spectral entropy. Notably, we first present the method for estimating graph spectral entropy without the eigen-decomposition of Laplacian matrix and provide the theoretical upper error bound. A maximum entropy regularization then acts in the latent space, which directly increases the graph spectral entropy. Extensive experiments show that MEGAE outperforms all the other state-of-the-art imputation methods on a variety of benchmark datasets.

Adaptive Maximum Entropy Graph-Guided Fast Locality Discriminant Analysis.

Linear discriminant analysis (LDA) aims to find a low-dimensional space in which data points in the same class are to be close to each other while keeping data points from different classes apart. To improve the robustness of LDA to non-Gaussian distribution data, most existing discriminant analysis methods extend LDA by approximating the underlying manifold of data. However, these methods suffer from the following problems: 1) local affinity or reconstruction coefficients are learned on the basis of the relationships of all data pairs, which would lead to a sharp increase in the amount of computation and 2) they learn the manifold information in the original space, ignoring the interference of the noise and redundant features. Motivated by these challenges, this article represents a novel discriminant analysis model, called fast and adaptive locality discriminant analysis (FALDA), to improve the efficiency and robustness. First, with the anchor-based strategy, a bipartite graph of each class is constructed to characterize the local structure of data. Since the number of anchor points is far less than that of data points, learning of fuzzy membership relationships between data points and anchor points within each class can save training time. Second, a maximum entropy regularization is introduced to control the uniformity of the weights of graphs and avoid the trivial solution. Third, the above relationships are updated adaptively in the process of dimensionality reduction, which can suppress the interference of the noise and redundant features. Fourth, the whitening constraint is imposed on the projection matrix to remove the relevance between features and restrict the total scatter of data in the subspace. Last but not the least, data with complex distribution can be explicitly divided into sub-blocks according to the learned anchor points (or subclass center points). We test our proposed method on synthetic data, benchmark datasets, and imbalanced datasets. Promising experimental results demonstrate the success of this novel model.

Transfer Learning-Based Collaborative Multiview Clustering

Collaborative multiview clustering methods can efficiently realize the view fusion by exploring complementary and consistent information among multiple views. However, these studies ignore all the differences between multiple views in fusion. In fact, in the multiview clustering, the data are diverse from view to view. The larger the difference between any two views is, the more the fusion of these views is required. Moreover, a global tradeoff parameter is generally adopted to restrain the penalty related to the disagreement of all views, which is often defined empirically. Inspired by the idea of transfer learning, a series of novel collaborative multiview clustering algorithms are proposed to tackle these challenges. In the most basic one, each view performs clustering independently and learns from others to improve its own clustering performance, in which a global learning factor is defined to control the interaction between multiple views. The fuzzy memberships are regarded as the important knowledge to provide guidance between views, and the consensus constraint is defined to ensure the consistent partitions of all views. In addition, the local adaptive learning factors between any two views instead of a global fixed one are adopted in an improved version to emphasize the difference between views, and the adjustment strategy for the learning factor is further designed to guarantee the stability of multiview clustering without the influence of initial values. Finally, to identify the significance of different views to the clustering, the extended versions are excavated with the assignment of view weights and the maximum entropy regularization technique is employed to optimize the weights. Experiments on various real-world multiview datasets verify the superiority of the presented approaches.

A Novel Combined Regularization for Identifying Random Load Sources on Coal‐Rock Structure

This study discusses the practical engineering problem of determining random load sources on coal‐rock structures. A novel combined regularization technique combining mollification method (MM) and discrete regularization (DR), which was called MM‐DR technique, was proposed to reconstruct random load sources on coal‐rock structures. MM‐DR technology is compared with DR, Tikhonov regularization (TR), and maximum entropy regularization (MER) in load reconstruction. The results show that the reconstructed random load sources are more consistent with the real load sources using MM‐DR technique combined with particle swarm optimization (PSO) and L‐curve method, which was named as PSO‐L method, and selecting optimal ω value of kernel function is beneficial to overcome the ill‐posed of random load sources reconstruction and to obtain the stable and approximate solutions. The method proposed in this study provides a theoretical basis for the recognition of coal‐rock interface.

Maximum Entropy Technique and Regularization Functional for Determining the Pharmacokinetic Parameters in DCE-MRI

This paper aims to solve the arterial input function (AIF) determination in dynamic contrast-enhanced MRI (DCE-MRI), an important linear ill-posed inverse problem, using the maximum entropy technique (MET) and regularization functionals. In addition, estimating the pharmacokinetic parameters from a DCE-MR image investigations is an urgent need to obtain the precise information about the AIF–the concentration of the contrast agent on the left ventricular blood pool measured over time. For this reason, the main idea is to show how to find a unique solution of linear system of equations generally in the form of y=Ax+b, named an ill-conditioned linear system of equations after discretization of the integral equations, which appear in different tomographic image restoration and reconstruction issues. Here, a new algorithm is described to estimate an appropriate probability distribution function for AIF according to the MET and regularization functionals for the contrast agent concentration when applying Bayesian estimation approach to estimate two different pharmacokinetic parameters. Moreover, by using the proposed approach when analyzing simulated and real datasets of the breast tumors according to pharmacokinetic factors, it indicates that using Bayesian inference—that infer the uncertainties of the computed solutions, and specific knowledge of the noise and errors—combined with the regularization functional of the maximum entropy problem, improved the convergence behavior and led to more consistent morphological and functional statistics and results. Finally, in comparison to the proposed exponential distribution based on MET and Newton’s method, or Weibull distribution via the MET and teaching–learning-based optimization (MET/TLBO) in the previous studies, the family of Gamma and Erlang distributions estimated by the new algorithm are more appropriate and robust AIFs.

Hyperspectral Imagery Classification via Random Multigraphs Ensemble Learning

Hyperspectral imagery (HSI) classification, which attempts to assign hyperspectral pixels with proper labels, has drawn significant attention in various applications. Recently, the graph-based semisupervised learning (SSL) approaches have shown the outstanding ability to handle the situation of limited labeled data for classification task. However, it is hard to construct the pairwise adjacent graph due to the high dimensionality of hyperspectral data. Besides, the graph-based SSL models are usually decided by a single classifier, which fail to effectively learn the complex structures and intrinsic properties of HSI. To address these problems, we propose a novel graph-based SSL classification model for HSI, which is based on random multigraphs construction and ensemble strategy (RMGE). Specifically, the anchor graph (AG) is constructed with spatial–spectral features, which integrates the spatial characteristics extracted by local binary pattern on each selected spectrum, preserving fine structures of local region. In order to enhance the discriminative capability of the classifier and avoid the trivial solution, the maximum entropy regularization is added into adjacent AG model. In addition, to capture the diversity of HSI data effectively, we design the ensemble framework by employing multiple AGs to learn HSI features. Experiments conducted on real hyperspectral datasets indicate that the proposed RMGE shows better performance than that of state-of-the-art approaches.

Regularizing Attention Networks for Anomaly Detection in Visual Question Answering

For stability and reliability of real-world applications, the robustness of DNNs in unimodal tasks has been evaluated. However, few studies consider abnormal situations that a visual question answering (VQA) model might encounter at test time after deployment in the real-world. In this study, we evaluate the robustness of state-of-the-art VQA models to five different anomalies, including worst-case scenarios, the most frequent scenarios, and the current limitation of VQA models. Different from the results in unimodal tasks, the maximum confidence of answers in VQA models cannot detect anomalous inputs, and post-training of the outputs, such as outlier exposure, is ineffective for VQA models. Thus, we propose an attention-based method, which uses confidence of reasoning between input images and questions and shows much more promising results than the previous methods in unimodal tasks. In addition, we show that a maximum entropy regularization of attention networks can significantly improve the attention-based anomaly detection of the VQA models. Thanks to the simplicity, attention-based anomaly detection and the regularization are model-agnostic methods, which can be used for various cross-modal attentions in the state-of-the-art VQA models. The results imply that cross-modal attention in VQA is important to improve not only VQA accuracy, but also the robustness to various anomalies.

Local structured feature learning with dynamic maximum entropy graph

In recent years, Linear Discriminant Analysis (LDA) has seen huge adoption in data mining applications. Due to its globality, it is incompetent to handle multimodal data. Besides, most of LDA’s variants learn the projection matrix based on the pre-defined similarity matrix, which is easily affected by noisy and irrelevant features. To address above two issues, a novel local structured feature learning with Dynamic Maximum Entropy Graph (DMEG) method is developed which firstly develops a more discriminative LDA with whitening constraint that can minimize the within-class scatter while keeping the total samples scatter unchanged simultaneously. Second, for exploring the local structure of data, the ℓ0-norm constraint is imposed on similarity matrix to ensure the k connectivity on graph. More importantly, proposed model learns the similarity and projection matrix simultaneously to ensure that the neighborships can be found in the optimal subspace where the noise have been removed already. Moreover, a maximum entropy regularization is employed to reinforce the discriminability of graph and avoid the trivial solution. Last but not least, an efficient iterative optimization algorithm is provided to optimize proposed model with a NP-hard constraint. Extensive experiments conducted on synthetic and several real-world data sets demonstrate the efficiency in classification task and robustness to noise of proposed method.

Non-parametric density reconstruction of the Galactic bulge area using red clump stars in the VVV survey

ABSTRACT Studies of the red clump giant population in the inner Milky Way suggest the Galactic bulge/bar has a boxy/peanut/X-shaped structure as predicted by its formation via a disc buckling instability. We used a non-parametric method of estimating the Galactic bulge morphology that is based on maximum entropy regularization. This enabled us to extract the 3D distribution of the red giant stars in the bulge from deep photometric catalogues of the VISTA Variables in the Via Lactea survey. Our high-resolution reconstruction confirms the well-known boxy/peanut/X-shaped structure of the bulge. We also find spiral arm structures that extend to around 3 kpc in front of and behind the bulge and are on different sides of the bulge major axis. We show that the detection of these structures is robust to the uncertainties in the luminosity function.

Image reconstruction from undersampled confocal microscopy data using multiresolution based maximum entropy regularization

We consider the problem of reconstructing 2D images from randomly under-sampled confocal microscopy samples. The well known and widely celebrated total variation regularization, which is the ℓ1 norm of derivatives, turns out to be unsuitable for this problem; it is unable to handle both noise and under-sampling together. This issue is linked with the notion of phase transition phenomenon observed in compressive sensing research, which is essentially the break-down of total variation methods, when sampling density gets lower than certain threshold. The severity of this breakdown is determined by the so-called mutual incoherence between the derivative operators and measurement operator. In our problem, the mutual incoherence is low, and hence the total variation regularization gives serious artifacts in the presence of noise even when the sampling density is not very low. There has been very few attempts in developing regularization methods that perform better than total variation regularization for this problem. We develop a multi-resolution based regularization method that is adaptive to image structure. In our approach, the desired reconstruction is formulated as a series of coarse-to-fine multi-resolution reconstructions; for reconstruction at each level, the regularization is constructed to be adaptive to the image structure, where the information for adaption is obtained from the reconstruction obtained at coarser resolution level. This adaptation is achieved by using maximum entropy principle, where the required adaptive regularization is determined as the maximizer of entropy subject to the information extracted from the coarse reconstruction as constraints. We also utilize the directionally adaptive second order derivatives for constructing the regularization with directions guided by the given coarse reconstruction, which leads to an improved suppression of artifacts. We demonstrate the superiority of the proposed regularization method over existing ones using several reconstruction examples.

A Novel Improved Maximum Entropy Regularization Technique and Application to Identification of Dynamic Loads on the Coal Rock

A new signal processing algorithm was proposed to identify the dynamic load acting on the coal-rock structure. First, the identification model for dynamic load is established through the relationship between the uncertain load vector, and the assembly matrix of the responses was measured by the machinery dynamic system. Then, the entropy item of maximum entropy regularization (MER) is redesigned using the robust estimation method, and the elongated penalty function according to the ill-posedness characteristics of load identification, which was named as a novel improved maximum entropy regularization (IMER) technique, was proposed to process the dynamic load signals. Finally, the load identification problem is transformed into an unconstrained optimization problem and an improved Newton iteration algorithm was proposed to solve the objective function. The result of IMER technique is compared with MER technique, and it is found that IMER technique is available for analyzing the dynamic load signals due to higher signal-noise ratio, lower restoration time, and fewer iterative steps. Experiments were performed to investigate the effect on the performance of dynamic load signals identification by different regularization parameters and calculation parameters, pi, respectively. Experimental results show that the identified dynamic load signals are closed to the actual load signals using IMER technique combined with the proposed PSO-L regularization parameter selection method. Selecting optimal calculated parameters pi is helpful to overcome the ill-condition of dynamic load signals identification and to obtain the stable and approximate solutions of inverse problems in practical engineering. Meanwhile, the proposed IMER technique can also play a guiding role for the coal-rock interface identification.

A new method for NMR data inversion based on double-parameter regularization

Nuclear magnetic resonance (NMR) technology plays a significant role in petroleum exploration. NMR data can be processed using inversion methods to reflect the relaxation information of all the components. We have developed a new double-parameter regularization (DPR) method for the inversion of NMR data, whose regularization terms consist of Tikhonov regularization and maximum entropy regularization. The objective function for the DPR method was solved using the Levenberg-Marquardt method, the proportional coefficient of the regularization parameter was obtained using an iteration procedure, and the optimum regularization parameter of the DPR method was selected using an S-curve. The relationship between the optimum regularization parameter and the signal-to-noise ratio (S/N) of the data was evaluated. Moreover, we compared the results of the NMR inversion obtained from the norm smooth method, the maximum entropy method, and the DPR method for simulated data. We evaluated how the proportional coefficient of the regularization parameter affected the inverted [Formula: see text] distributions and processed field NMR log data for a tight sandstone reservoir using the DPR method. The results indicated that the optimum regularization parameter for the DPR method gradually decreases with increasing data S/N. The accuracy is higher for the DPR method than for the norm smooth method and the maximum entropy method under low-S/N conditions. It is of great importance to select the proportional coefficient for the DPR method. The inverted [Formula: see text] distributions are similar for the DPR method and the norm smooth method when the proportional coefficient is small, and this is similar for the DPR method and the maximum entropy method when the proportional coefficient is large.

Identification of Random Dynamic Force Using an Improved Maximum Entropy Regularization Combined with a Novel Conjugate Gradient

We propose a novel mathematical algorithm to offer a solution for the inverse random dynamic force identification in practical engineering. Dealing with the random dynamic force identification problem using the proposed algorithm, an improved maximum entropy (IME) regularization technique is transformed into an unconstrained optimization problem, and a novel conjugate gradient (NCG) method was applied to solve the objective function, which was abbreviated as IME‐NCG algorithm. The result of IME‐NCG algorithm is compared with that of ME, ME‐CG, ME‐NCG, and IME‐CG algorithm; it is found that IME‐NCG algorithm is available for identifying the random dynamic force due to smaller root mean‐square‐error (RMSE), lower restoration time, and fewer iterative steps. Example of engineering application shows that L‐curve method is introduced which is better than Generalized Cross Validation (GCV) method and is applied to select regularization parameter; thus the proposed algorithm can be helpful to alleviate the ill‐conditioned problem in identification of dynamic force and to acquire an optimal solution of inverse problem in practical engineering.

Quantifying Trace Amounts of Aggregates in Biopharmaceuticals Using Analytical Ultracentrifugation Sedimentation Velocity: Bayesian Analyses and F Statistics.

Analytical ultracentrifugation-sedimentation velocity (AUC-SV) is often used to quantify high molar mass species (HMMS) present in biopharmaceuticals. Although these species are often present in trace quantities, they have received significant attention due to their potential immunogenicity. Commonly, AUC-SV data is analyzed as a diffusion-corrected, sedimentation coefficient distribution, or c(s), using SEDFIT to numerically solve Lamm-type equations. SEDFIT also utilizes maximum entropy or Tikhonov-Phillips regularization to further allow the user to determine relevant sample information, including the number of species present, their sedimentation coefficients, and their relative abundance. However, this methodology has several, often unstated, limitations, which may impact the final analysis of protein therapeutics. These include regularization-specific effects, artificial "ripple peaks," and spurious shifts in the sedimentation coefficients. In this investigation, we experimentally verified that an explicit Bayesian approach, as implemented in SEDFIT, can largely correct for these effects. Clear guidelines on how to implement this technique and interpret the resulting data, especially for samples containing micro-heterogeneity (e.g., differential glycosylation), are also provided. In addition, we demonstrated how the Bayesian approach can be combined with F statistics to draw more accurate conclusions and rigorously exclude artifactual peaks. Numerous examples with an antibody and an antibody-drug conjugate were used to illustrate the strengths and drawbacks of each technique.

GNSS‐ISR data fusion: General framework with application to the high‐latitude ionosphere

AbstractA mathematical framework is presented for the fusion of electron density measured by incoherent scatter radar (ISR) and total electron content (TEC) measured using global navigation satellite systems (GNSS). Both measurements are treated as projections of an unknown density field (for GNSS‐TEC the projection is tomographic; for ISR the projection is a weighted average over a local spatial region) and discrete inverse theory is applied to obtain a higher fidelity representation of the field than could be obtained from either modality individually. The specific implementation explored herein uses the interpolated ISR density field as initial guess to the combined inverse problem, which is subsequently solved using maximum entropy regularization. Simulations involving a dense meridional network of GNSS receivers near the Poker Flat ISR demonstrate the potential of this approach to resolve sub‐beam structure in ISR measurements. Several future directions are outlined, including (1) data fusion using lower level (lag product) ISR data, (2) consideration of the different temporal sampling rates, (3) application of physics‐based regularization, (4) consideration of nonoptimal observing geometries, and (5) use of an ISR simulation framework for optimal experiment design.

A nonparametric clustering algorithm with a quantile-based likelihood estimator.

Clustering is a representative of unsupervised learning and one of the important approaches in exploratory data analysis. By its very nature, clustering without strong assumption on data distribution is desirable. Information-theoretic clustering is a class of clustering methods that optimize information-theoretic quantities such as entropy and mutual information. These quantities can be estimated in a nonparametric manner, and information-theoretic clustering algorithms are capable of capturing various intrinsic data structures. It is also possible to estimate information-theoretic quantities using a data set with sampling weight for each datum. Assuming the data set is sampled from a certain cluster and assigning different sampling weights depending on the clusters, the cluster-conditional information-theoretic quantities are estimated. In this letter, a simple iterative clustering algorithm is proposed based on a nonparametric estimator of the log likelihood for weighted data sets. The clustering algorithm is also derived from the principle of conditional entropy minimization with maximum entropy regularization. The proposed algorithm does not contain a tuning parameter. The algorithm is experimentally shown to be comparable to or outperform conventional nonparametric clustering methods.

Core surface magnetic field evolution 2000-2010

SUMMARY We present new dedicated core surface field models spanning the decade from 2000.0 to 2010.0. These models, called gufm-sat, are based on CHAMP, Orsted and SAC-C satellite observations along with annual differences of processed observatory monthly means. A spatial parametrization of spherical harmonics up to degree and order 24 and a temporal parametrization of sixth-order B-splines with 0.25 yr knot spacing is employed. Models were constructed by minimizing an absolute deviation measure of misfit along with measures of spatial and temporal complexity at the core surface. We investigate traditional quadratic or maximum entropy regularization in space, and second or third time derivative regularization in time. Entropy regularization allows the construction of models with approximately constant spectral slope at the core surface, avoiding both the divergence characteristic of the crustal field and the unrealistic rapid decay typical of quadratic regularization at degrees above 12. We describe in detail aspects of the models that are relevant to core dynamics. Secular variation and secular acceleration are found to be of lower amplitude under the Pacific hemisphere where the core field is weaker. Rapid field evolution is observed under the eastern Indian Ocean associated with the growth and drift of an intense low latitude flux patch. We also find that the present axial dipole decay arises from a combination of subtle changes in the southern hemisphere field morphology.

A multiscale approach to the elastic moduli of biomembrane networks

We develop equilibrium fluctuation formulae for the isothermal elastic moduli of discrete biomembrane models at different scales. We account for the coupling of large stretching and bending strains of triangulated network models endowed with harmonic and dihedral angle potentials, on the basis of the discrete-continuum approach presented in Schmidt and Fraternali (J Mech Phys Solids 60:172-180, 2012). We test the proposed equilibrium fluctuation formulae with reference to a coarse-grained molecular dynamics model of the red blood cell (RBC) membrane (Marcelli et al. in Biophys J 89:2473-2480, 2005; Hale et al. in Soft Matter 5:3603-3606, 2009), employing a local maximum-entropy regularization of the fluctuating configurations (Fraternali et al. in J Comput Phys 231:528-540, 2012). We obtain information about membrane stiffening/softening due to stretching, curvature, and microscopic undulations of the RBC model. We detect local dependence of the elastic moduli over the RBC membrane, establishing comparisons between the present theory and different approaches available in the literature.