Entropy Regularization Research Articles

Bezdek-Type Fuzzified Co-Clustering Algorithm

In this study, two co-clustering algorithms based on Bezdek-type fuzzification of fuzzy clustering are proposed for categorical multivariate data. The two proposed algorithms are motivated by the fact that there are only two fuzzy co-clustering methods currently available – entropy regularization and quadratic regularization – whereas there are three fuzzy clustering methods for vectorial data: entropy regularization, quadratic regularization, and Bezdek-type fuzzification. The first proposed algorithm forms the basis of the second algorithm. The first algorithm is a variant of a spherical clustering method, with the kernelization of a maximizing model of Bezdek-type fuzzy clustering with multi-medoids. By interpreting the first algorithm in this way, the second algorithm, a spectral clustering approach, is obtained. Numerical examples demonstrate that the proposed algorithms can produce satisfactory results when suitable parameter values are selected.

Spectral blind deconvolution with differential entropy regularization for infrared spectrum

Estimating an accuracy spectrum from a linearly degraded and noisy data record is often desirable in infrared spectroscopy application. In this paper, we proposed a new framework which can estimate the desired spectrum and the instrument function simultaneously from the degraded data. Firstly, the instrument function is parametrically represented as a Lorentzian function for the first time. Then, we construct the cost functional with the DE regularization, and minimize the functional to obtain the desired spectrum and instrument function. It has been found that the program is very useful to determine accurate line widths and peak positions from degraded spectral spectrum. Comparative results including quantitative and qualitative analyses manifest that the proposed method outperforms the conventional methods on peak narrowing and noise suppression.

Regularization schemes for minimum error entropy principle

We introduce a learning algorithm for regression generated by a minimum error entropy (MEE) principle and regularization schemes in reproducing kernel Hilbert spaces. This empirical MEE algorithm is highly related to a scaling parameter arising from Parzen windowing. The purpose of this paper is to carry out consistency analysis when the scaling parameter is large. Explicit learning rates are provided. Novel approaches are proposed to overcome the difficulties in bounding the output function uniformly and in the special MEE feature that the regression function may not be a minimizer of the error entropy.

A stable approach for model order selection in nonnegative matrix factorization

In order to find the correct model order in non-negative matrix factorization (NMF), an algorithm called automatic relevance determination (ARD) is proposed in Tan and Fevotte (2013). The algorithm explores the similarities of the NMF components and removes redundant ones iteratively. However, the algorithm can yield over-parsimonious representations where ground truth patterns can be grouped into one single component to cause superposition. In this paper, mixed entropy regularized NMF (MER-NMF) is proposed to overcome the above problem. In MER-NMF, the objective function of NMF is regularized by minimizing a mixed entropy of the coefficient matrix which is a weighted sum of two parts: the entropy of all the entries and the entropy of the row sums of the coefficient matrix. With the mixed entropy regularization, the algorithm tends to yield sharper activations of the components for each sample. By combining MER-NMF and ARD-NMF, correct number of components can always be selected according to our experiments.

A nonparametric clustering algorithm with a quantile-based likelihood estimator.

Clustering is a representative of unsupervised learning and one of the important approaches in exploratory data analysis. By its very nature, clustering without strong assumption on data distribution is desirable. Information-theoretic clustering is a class of clustering methods that optimize information-theoretic quantities such as entropy and mutual information. These quantities can be estimated in a nonparametric manner, and information-theoretic clustering algorithms are capable of capturing various intrinsic data structures. It is also possible to estimate information-theoretic quantities using a data set with sampling weight for each datum. Assuming the data set is sampled from a certain cluster and assigning different sampling weights depending on the clusters, the cluster-conditional information-theoretic quantities are estimated. In this letter, a simple iterative clustering algorithm is proposed based on a nonparametric estimator of the log likelihood for weighted data sets. The clustering algorithm is also derived from the principle of conditional entropy minimization with maximum entropy regularization. The proposed algorithm does not contain a tuning parameter. The algorithm is experimentally shown to be comparable to or outperform conventional nonparametric clustering methods.

Information-theoretic semi-supervised metric learning via entropy regularization.

We propose a general information-theoretic approach to semi-supervised metric learning called SERAPH (SEmi-supervised metRic leArning Paradigm with Hypersparsity) that does not rely on the manifold assumption. Given the probability parameterized by a Mahalanobis distance, we maximize its entropy on labeled data and minimize its entropy on unlabeled data following entropy regularization. For metric learning, entropy regularization improves manifold regularization by considering the dissimilarity information of unlabeled data in the unsupervised part, and hence it allows the supervised and unsupervised parts to be integrated in a natural and meaningful way. Moreover, we regularize SERAPH by trace-norm regularization to encourage low-dimensional projections associated with the distance metric. The nonconvex optimization problem of SERAPH could be solved efficiently and stably by either a gradient projection algorithm or an EM-like iterative algorithm whose M-step is convex. Experiments demonstrate that SERAPH compares favorably with many well-known metric learning methods, and the learned Mahalanobis distance possesses high discriminability even under noisy environments.

Anomalous Small Angle X-Ray Scattering Simulations: Proof of Concept for Distance Measurements for Nanoparticle-Labelled Biomacromolecules in Solution

Anomalous small angle X-ray scattering can in principle be used to determine distances between metal label species on biological molecules. Previous experimental studies in the past were unable to distinguish the label-label scattering contribution from that of the molecule, because of the use of atomic labels; these labels contribute only a small proportion of the total scattering signal. However, with the development of nanocrystal labels (of 50–100 atoms) there is the possibility for a renewed attempt at applying anomalous small angle X-ray scattering for distance measurement. This is because the contribution to the scattered signal is necessarily considerably stronger than for atomic labels. Here we demonstrate through simulations, the feasibility of the technique to determine the end-to-end distances of labelled nucleic acid molecules as well as other internal distances mimicking a labelled DNA binding protein if the labels are dissimilar metal nanocrystals. Of crucial importance is the ratio of mass of the nanocrystals to that of the labelled macromolecule, as well as the level of statistical errors in the scattering intensity measurements. The mathematics behind the distance determination process is presented, along with a fitting routine than incorporates maximum entropy regularisation.

Bias Correction for Magnetic Resonance Images via Joint Entropy Regularization

Due to the imperfections of the radio frequency (RF) coil or object-dependent electrodynamic interactions, magnetic resonance (MR) images often suffer from a smooth and biologically meaningless bias field, which causes severe troubles for subsequent processing and quantitative analysis. To effectively restore the original signal, this paper simultaneously exploits the spatial and gradient features of the corrupted MR images for bias correction via the joint entropy regularization. With both isotropic and anisotropic total variation (TV) considered, two nonparametric bias correction algorithms have been proposed, namely IsoTVBiasC and AniTVBiasC. These two methods have been applied to simulated images under various noise levels and bias field corruption and also tested on real MR data. The test results show that the proposed two methods can effectively remove the bias field and also present comparable performance compared to the state-of-the-art methods.

Entropic Approach for Emission Rate Estimation of Area Pollutant Sources

The estimat ion of the area source pollutant strength is a relevant issue for atmospheric environ ment. Th is characterizes an inverse problem in the atmospheric pollution dispersion. In the inverse analysis, an area source domain is considered, where the strength of such area source term is assumed unknown. The inverse problem is formu lated as a non-linear optimization approach, whose objective function is given by the square difference between the measured pollutant concentration and the mathematical models, associated with a regularization operator. The forward problem is addressed by a source-receptor scheme, where a regressive Lagrangian model is applied to co mpute the transition matrix. A quasi-Newton method is emp loyed for minimizing the object ive function. The second order maximu m entropy regularizat ion is used, and the regularizat ion parameter is calculated by the L-curve technique. This inverse problem methodology is tested with synthetic observational data, producing good inverse solutions.

Dual Exclusive Partition in Fuzzy CoDoK and SCAD-based Fuzzy Co-clustering

Fuzzy co-clustering is a basic technique for revealing intrinsic co-cluster structures from cooccurrence information among objects and items. In most of fuzzy co-clustering algorithms, objects and items are partitioned based on different constraints. Objects are forced to be exclusively partitioned like as Fuzzy c-Means (FCM), while item memberships often represent just the relative significance of items in each cluster, i.e., items can be shared by multiple clusters. In a previous work, exclusive partition of items were achieved by introducing a penalty term in Fuzzy Clustering for Categorical Multivariate data (FCCM), which is an FCM-type co-clustering with entropy regularization mechanism. In this paper, the applicability of dual exclusive partition of objects and items are discussed in the frameworks of Fuzzy CoDoK and SCAD-based fuzzy co-clustering.

Semi-supervised fuzzy clustering with metric learning and entropy regularization

Existing methods for semi-supervised fuzzy c-means (FCMs) suffer from the following issues: (1) the Euclidean distance tends to work poorly if each feature of the instance is unequal variance as well as correlation from others and (2) it is generally uneasy to assign an appropriate value for the parameter m involved in their objective function. To address these problems, we develop a novel semi-supervised metric-based fuzzy clustering algorithm called SMUC by introducing metric learning and entropy regularization simultaneously into the conventional fuzzy clustering algorithm. More specifically, SMUC focuses on learning a Mahalanobis distance metric from side information given by the user to displace the Euclidean distance in FCM-based methods. Thus, it has the same flavor as typical supervised metric algorithms, which makes the distance between instances within a cluster smaller than that between instances belonging to different clusters. Moreover, SMUC introduces maximum entropy as a regularized term in its objective function such that its resulting formulas have the clear physical meaning compared with the other semi-supervised FCM methods. In addition, it naturally avoids the choice on the parameter m due to such a maximum-entropy regularizer. The experiments on real-world data sets show the feasibility and effectiveness of the proposed method with encouraging results.

Core surface magnetic field evolution 2000-2010

SUMMARY We present new dedicated core surface field models spanning the decade from 2000.0 to 2010.0. These models, called gufm-sat, are based on CHAMP, Orsted and SAC-C satellite observations along with annual differences of processed observatory monthly means. A spatial parametrization of spherical harmonics up to degree and order 24 and a temporal parametrization of sixth-order B-splines with 0.25 yr knot spacing is employed. Models were constructed by minimizing an absolute deviation measure of misfit along with measures of spatial and temporal complexity at the core surface. We investigate traditional quadratic or maximum entropy regularization in space, and second or third time derivative regularization in time. Entropy regularization allows the construction of models with approximately constant spectral slope at the core surface, avoiding both the divergence characteristic of the crustal field and the unrealistic rapid decay typical of quadratic regularization at degrees above 12. We describe in detail aspects of the models that are relevant to core dynamics. Secular variation and secular acceleration are found to be of lower amplitude under the Pacific hemisphere where the core field is weaker. Rapid field evolution is observed under the eastern Indian Ocean associated with the growth and drift of an intense low latitude flux patch. We also find that the present axial dipole decay arises from a combination of subtle changes in the southern hemisphere field morphology.

Regularizing made-to-measure particle models of galaxies

Made-to-measure methods such as the parallel code NMAGIC are powerful tools to build galaxy models reproducing observational data. They work by adapting the particle weights in an N-body system until the target observables are well matched. Here we introduce a moving prior regularization (MPR) method for such particle models. It is based on determining from the particles a distribution of priors in phase-space, which are updated in parallel with the weight adaptation. This method allows one to construct smooth models from noisy data without erasing global phase-space gradients. We first apply MPR to a spherical system for which the distribution function can in theory be uniquely recovered from idealized data. We show that NMAGIC with MPR indeed converges to the true solution with very good accuracy, independent of the initial particle model. Compared to the standard weight entropy regularization, biases in the anisotropy structure are removed and local fluctuations in the intrinsic distribution function are reduced. We then investigate how the uncertainties in the inferred dynamical structure increase with less complete and noisier kinematic data, and how the dependence on the initial particle model also increases. Finally, we apply the MPR technique to the two intermediate-luminosity elliptical galaxies NGC 4697 and NGC 3379, obtaining smoother dynamical models in luminous and dark matter potentials.

On Entropy Based Fuzzy Non Metric Model – Proposal, Kernelization and Pairwise Constraints –

The fuzzy non metric model is a kind of clustering method in which belongingness or the membership grade of each datum to each cluster is calculated directly from dissimilarities between data, and cluster centers are not used. In this paper, we first construct a new fuzzy non metric model with entropy regularization. Second, we kernelize the proposed method by introducing kernel functions. Third, we consider pairwise constraints with the proposed method. We then confirm the above methods through some simple numerical examples.

Comparison between method of moments and entropy regularization algorithm applied to parameter estimation for mixed-Weibull distribution

Mixed-Weibull distribution has been used to model a wide range of failure data sets, and in many practical situations the number of components in a mixture model is unknown. Thus, the parameter estimation of a mixed-Weibull distribution is considered and the important issue of how to determine the number of components is discussed. Two approaches are proposed to solve this problem. One is the method of moments and the other is a regularization type of fuzzy clustering algorithm. Finally, numerical examples and two real data sets are given to illustrate the features of the proposed approaches.

Regularization methods for the extraction of depth profiles from simulated ARXPS data derived from overlayer/substrate models

Starting from posited input depth profiles of silicon oxide on silicon, 100 sets of noisy simulated ARXPS data were created for each oxide layer thickness of 3, 6, 9, 12, 15, 18, 21, 24 and 27 Å. Oxygen depth profiles were then recovered from the noisy simulated data using regularized inversion methods, including maximum entropy and Tikhonov regularization. Three regularization parameters were used: one determined by the S-curve method, one determined by the L-curve method and a third corresponding to the closest correspondence between the input and extracted profiles. The various regularization schemes evaluated were ranked with respect to their ability to reproduce the input profile.

Online variance minimization

We consider the following type of online variance minimization problem: In every trial t our algorithms get a covariance matrix C t and try to select a parameter vector w t?1 such that the total variance over a sequence of trials $\sum_{t=1}^{T} (\boldsymbol {w}^{t-1})^{\top} \boldsymbol {C}^{t}\boldsymbol {w}^{t-1}$ is not much larger than the total variance of the best parameter vector u chosen in hindsight. Two parameter spaces in ? n are considered--the probability simplex and the unit sphere. The first space is associated with the problem of minimizing risk in stock portfolios and the second space leads to an online calculation of the eigenvector with minimum eigenvalue of the total covariance matrix $\sum_{t=1}^{T} \boldsymbol {C}^{t}$ . For the first parameter space we apply the Exponentiated Gradient algorithm which is motivated with a relative entropy regularization. In the second case, the algorithm has to maintain uncertainty information over all unit directions u. For this purpose, directions are represented as dyads uu ? and the uncertainty over all directions as a mixture of dyads which is a density matrix. The motivating divergence for density matrices is the quantum version of the relative entropy and the resulting algorithm is a special case of the Matrix Exponentiated Gradient algorithm. In each of the two cases we prove bounds on the additional total variance incurred by the online algorithm over the best offline parameter.

Deriving the radial-velocity variations induced by stellar activity from high-precision photometry

Stellar activity induces apparent radial velocity (RV) variations in late-type main-sequence stars that may hamper the detection of low-mass planets and the measurement of their mass. We use simultaneous measurements of the active planet host star HD189733 with high-precision optical photometry by the MOST satellite and high-resolution spectra by SOPHIE. We apply on this unique dataset a spot model to predict the activity-induced RV variations and compare them with the observed ones. The model is based on the rotational modulation of the stellar flux. A maximum entropy regularization is applied to find a unique and stable solution for the distribution of the active regions versus stellar longitude. The RV variations are synthesized considering the effects on the line profiles of the brightness perturbations due to dark spots and bright faculae and the reduction of the convective blueshifts in the active regions. The synthesized RV time series shows a remarkably good agreement with the observed one although variations on timescales shorter than 4-5 days cannot be reproduced by our model. Persistent active longitudes are revealed by the spot modelling. They rotate with slightly different periods yielding a minimum relative amplitude of the differential rotation of \Delta \Omega / \Omega = 0.23 \pm 0.10. The method proves capable of reducing the power of the activity-induced RV variations by a factor from 2 to 10 at the rotation frequency and its harmonics up to the third. Thanks to the high-precision space-borne photometry delivered by CoRoT, Kepler, or later PLATO, it is possible to map the longitudinal distribution of active regions in late-type stars and apply the method presented in this paper to reduce remarkably the impact of stellar activity on their RV jitter allowing us to confirm the detection of low-mass planets or refine the measurement of their mass.

STEREOLOGICAL CORRECTION OF MINERAL LIBERATION GRADE DISTRIBUTIONS ESTIMATED BY SINGLE SECTIONING OF PARTICLES

The liberation distribution of ore samples is of considerable interest for process optimisation in the minerals industry. A scanning electron microscope-based automatic mineral analyser such as the LEO QEMSCAN system developed by CSIRO Minerals is a powerful tool for the estimation of linear or areal grade distributions of a population of ore particles based upon polished single particle sections. Stereological correction of a single section mineralogical grade distribution is recognised as an ill-posed inverse problem. The transformation kernel method with constrained entropy regularisation (King and Schneider, 1998) is adopted for the correction of stereological error in binary systems. An enhanced transformation kernel correction scheme is developed with an additional equality constraint for average grade as determined by section and volumetric sampling, in accordance with Delesse's fundamental stereological theorem. The usefulness of both correction methods is limited by the availability of kernels that appropriately model the relationship between volumetric and section grade distributions for the mineralogical sample of interest. The transformation kernel stereological correction methods are implemented in software available for use as part of the LEO QEMSCAN system. Both correction procedures are applied to areal section grade distributions of feed and concentrate from a mineral processing plant. The corrected grade distributions are in some instances found to be sensitive to the application of the average grade constraint. The statistical significance of differences in the corrected solutions is discussed.

The boundary structure in the analysis of reversibly interacting systems by sedimentation velocity

Sedimentation velocity (SV) experiments of heterogeneous interacting systems exhibit characteristic boundary structures that can usually be very easily recognized and quantified. For slowly interacting systems, the boundaries represent concentrations of macromolecular species sedimenting at different rates, and they can be interpreted directly with population models based solely on the mass action law. For fast reactions, migration and chemical reactions are coupled, and different, but equally easily discernable boundary structures appear. However, these features have not been commonly utilized for data analysis, for the lack of an intuitive and computationally simple model. The recently introduced effective particle theory (EPT) provides a suitable framework. Here, we review the motivation and theoretical basis of EPT, and explore practical aspects for its application. We introduce an EPT-based design tool for SV experiments of heterogeneous interactions in the software SEDPHAT. As a practical tool for the first step of data analysis, we describe how the boundary resolution of the sedimentation coefficient distribution c( s) can be further improved with a Bayesian adjustment of maximum entropy regularization to the case of heterogeneous interactions between molecules that have been previously studied separately. This can facilitate extracting the characteristic boundary features by integration of c( s). In a second step, these are assembled into isotherms as a function of total loading concentrations and fitted with EPT. Methods for addressing concentration errors in isotherms are discussed. Finally, in an experimental model system of alpha-chymotrypsin interacting with soybean trypsin inhibitor, we show that EPT provides an excellent description of the experimental sedimentation boundary structure of fast interacting systems.