Bregman Divergence Research Articles

Transfer learning for face recognition using fingerprint biometrics

Biometrics is a set of highly automated methods used for recognition purposes including analyzing physical traits of people and statistically measuring them. In forensic applications, fingerprint and face images are mostly used for recognition. There is a need during a criminal investigation to find out a face image of a person from its fingerprint. The proposed method able to identify the face of a person using an associated fingerprint. In this paper, Bregman divergence regularization is used to learn and optimize transferring subspace. This method first gleans knowledge from samples that are meant for training and transfer it to the testing samples. This regularization helps to minimize the probability distribution differences between two different domains. It helps to find a common subspace that boosts the performance of independent and identically distributed (i.i.d.) condition of samples. Practically the samples violate this i.i.d. condition. However, it will help to identify the correct suspect.

Pseudo-Riemannian geometry encodes information geometry in optimal transport

Optimal transport and information geometry both study geometric structures on spaces of probability distributions. Optimal transport characterizes the cost-minimizing movement from one distribution to another, while information geometry originates from coordinate invariant properties of statistical inference. Their relations and applications in statistics and machine learning have started to gain more attention. In this paper we give a new differential-geometric relation between the two fields. Namely, the pseudo-Riemannian framework of Kim and McCann, which provides a geometric perspective on the fundamental Ma–Trudinger–Wang (MTW) condition in the regularity theory of optimal transport maps, encodes the dualistic structure of statistical manifold. This general relation is described using the framework of c-divergence under which divergences are defined by optimal transport maps. As a by-product, we obtain a new information-geometric interpretation of the MTW tensor on the graph of the transport map. This relation sheds light on old and new aspects of information geometry. The dually flat geometry of Bregman divergence corresponds to the quadratic cost and the pseudo-Euclidean space, and the logarithmic L^{(alpha )}-divergence introduced by Pal and the first author has constant sectional curvature in a sense to be made precise. In these cases we give a geometric interpretation of the information-geometric curvature in terms of the divergence between a primal-dual pair of geodesics.

Myopic robust index tracking with Bregman divergence

Index tracking is a popular form of asset management. Typically, a quadratic function is used to define the tracking error of a portfolio and the look back approach is applied to solve the index tracking problem. We argue that a forward looking approach is more suitable, whereby the tracking error is expressed as an expectation of a function of the difference between the returns of the index and of the portfolio. We also assume that there is model uncertainty in the distribution of the assets, hence a robust version of the optimization problem needs to be adopted. We use Bregman divergence in describing the deviation between the nominal and actual (true) distribution of the components of the index. In this scenario, we derive the optimal robust index tracking portfolio in a semi-analytical form as a solution of a system of nonlinear equations. Several numerical results are presented that allow us to compare the performance of this robust portfolio with the optimal non-robust portfolio. We show that, especially during market downturns, the robust portfolio can be very advantageous.

Generalized Bregman Envelopes and Proximity Operators

Every maximally monotone operator can be associated with a family of convex functions, called the Fitzpatrick family or family of representative functions. Surprisingly, in 2017, Burachik and Martínez-Legaz showed that the well-known Bregman distance is a particular case of a general family of distances, each one induced by a specific maximally monotone operator and a specific choice of one of its representative functions. For the family of generalized Bregman distances, sufficient conditions for convexity, coercivity, and supercoercivity have recently been furnished. Motivated by these advances, we introduce in the present paper the generalized left and right envelopes and proximity operators, and we provide asymptotic results for parameters. Certain results extend readily from the more specific Bregman context, while others only extend for certain generalized cases. To illustrate, we construct examples from the Bregman generalizing case, together with the natural “extreme” cases that highlight the importance of which generalized Bregman distance is chosen.

Domain Adaptation via Bregman divergence minimization

In recent years, Fisher linear discriminant analysis (FLDA) based classifi-cation is one of the most successful approaches and has been shown effectiveperformance in different classification tasks. However, when the learning data(source domain) have a different distribution against the testing data (tar-get domain), the FLDA-based models may not be optimal, and the perfor-mance will be degraded, dramatically. To face this problem, in this paper, wepropose an optimal domain adaptation via Bregman divergence minimization(DAB) approach, in which the discriminative features of source and target do-mains are simultaneously learned via domain invariant representation. DABis designed based on the constraints of FLDA, with the objective to adaptthe coupled marginal and conditional distribution mismatches with Breg-man divergence minimization. Thus, the resulting representation can havewell functionality like FLDA and simultaneously have better discriminationability. Moreover, our proposed approach can be easily kernelized to dealwith nonlinear tasks. Extensive experiments on various benchmark datasetsshow that our DAB can effectively deal with the cross domain divergence andoutperforms several state-of-the-art domain adaptation approaches on cross-distribution domains.

Simplified Levenberg–Marquardt method in Banach spaces for nonlinear ill-posed operator equations

In 2011, Jin Qinian has proposed a Levenberg–Marquardt method, by making use of duality mapping and the Bregman distance, to get an approximate solution of a nonlinear ill-posed operator equation in Banach space using an a posteriori parameter choice strategy and Morozov-type stopping rule. The method considered by Jin Qinian was an extension of the method proposed by M. Hanke in 1997 for the Hilbert space case. In this paper, we suggest a modified variant of the method, namely, the simplified Levenberg–Marquardt scheme in Banach spaces. The advantage of the method considered in the paper is that, it is also applicable for the operator equation with non-smooth operator. We establish convergence of the method under a modified a posteriori parameter choice strategy which is more feasible than the one considered by Jin Qinian (2011). Numerical example to demonstrate the validity of the considered method is discussed.

An Objective Prior from a Scoring Rule.

In this paper, we introduce a novel objective prior distribution levering on the connections between information, divergence and scoring rules. In particular, we do so from the starting point of convex functions representing information in density functions. This provides a natural route to proper local scoring rules using Bregman divergence. Specifically, we determine the prior which solves setting the score function to be a constant. Although in itself this provides motivation for an objective prior, the prior also minimizes a corresponding information criterion.

A novel kernelized total Bregman divergence-based fuzzy clustering with local information for image segmentation

Kernel weighted fuzzy local information c-means clustering (KWFLICM) is of great significance in the field of fuzzy clustering. Nevertheless, it is sensitive to high noise or outliers. Aiming at the shortcoming of KWFLICM algorithm, this paper proposes a novel fuzzy local information c-means clustering algorithm utilizing total Bregman divergence driven by polynomial kernel function (TKWFLICM). Firstly, total Bregman divergence (TBD) is kernelized by global polynomial kernel function to solve the problem of kernel-based distance metric without rotation-invariance. Secondly, TBD with polynomial kernel metric is modified by local neighborhood information of current pixel to further enhance the ability to suppress the noise in image. In the end, the modified TBD with polynomial kernel is introduced into the objective function of KWFLICM, and then a novel robust fuzzy clustering with high robustness is obtained. Experimental results show that the proposed algorithm performs better than existing state-of-the-art KWFLICM algorithm, and the kernelized TBD can also be utilized in fuzzy clustering-related algorithms, which will enhance the anti-noise robustness of these algorithms.

Bregman-style Online Convex Optimization with Energy Harvesting Constraints

This paper considers online convex optimization (OCO) problems where decisions are constrained by available energy resources. A key scenario is optimal power control for an energy harvesting device with a finite capacity battery. The goal is to minimize a time-average loss function while keeping the used energy less than what is available. In this setup, the distribution of the randomly arriving harvestable energy (which is assumed to be i.i.d.) is unknown, the current loss function is unknown, and the controller is only informed by the history of past observations. A prior algorithm is known to achieve O(√T) regret by using a battery with an O(√T) capacity. This paper develops a new algorithm that maintains this asymptotic trade-off with the number of time steps T while improving dependency on the dimension of the decision vector from O(√n) to O(√log(n)). The proposed algorithm introduces a separation of the decision vector into amplitude and direction components. It uses two distinct types of Bregman divergence, together with energy queue information, to make decisions for each component.

Multi-block Bregman proximal alternating linearized minimization and its application to orthogonal nonnegative matrix factorization

We introduce and analyze BPALM and A-BPALM, two multi-block proximal alternating linearized minimization algorithms using Bregman distances for solving structured nonconvex problems. The objective function is the sum of a multi-block relatively smooth function (i.e., relatively smooth by fixing all the blocks except one) and block separable (nonsmooth) nonconvex functions. The sequences generated by our algorithms are subsequentially convergent to critical points of the objective function, while they are globally convergent under the KL inequality assumption. Moreover, the rate of convergence is further analyzed for functions satisfying the Łojasiewicz’s gradient inequality. We apply this framework to orthogonal nonnegative matrix factorization (ONMF) that satisfies all of our assumptions and the related subproblems are solved in closed forms, where some preliminary numerical results are reported.

Proximal-like incremental aggregated gradient method with Bregman distance in weakly convex optimization problems

Proximal-like incremental aggregated gradient method with Bregman distance in weakly convex optimization problems

A Bregman subgradient extragradient method with self-adaptive technique for solving variational inequalities in reflexive Banach spaces

In this paper, we introduce a self-adaptive Bregman subgradient extragradient method for solving variational inequalities in the framework of a reflexive Banach space. The step-adaptive strategy avoids the difficult task of choosing a stepsize based on the Lipschitz constant of the cost function of the variational inequalities and improves the performance of the algorithm. Moreover, the use of the Bregman distance technique allows the consideration of a general feasible set for the problem. Under some suitable conditions, we prove some weak and strong convergence results for the sequence generated by the algorithm without prior knowledge of the Lipschitz constant. We further provide an application to contact problems and some numerical experiments to illustrate the performance of the algorithm.

Variational regularization theory based on image space approximation rates

We present a new approach to convergence rate results for variational regularization. Avoiding Bregman distances and using image space approximation rates as source conditions we prove a nearly minimax theorem showing that the modulus of continuity is an upper bound on the reconstruction error up to a constant. Applied to Besov space regularization we obtain convergence rate results for 0, 2, q- and 0, p, p-penalties without restrictions on p, q ∈ (1, ∞). Finally we prove equivalence of Hölder-type variational source conditions, bounds on the defect of the Tikhonov functional, and image space approximation rates.

Testing the fit of relational models

Relational models generalize log-linear models to arbitrary discrete sample spaces by specifying effects associated with any subsets of their cells. A relational model may include an overall effect, pertaining to every cell after a reparameterization, and in this case, the properties of the maximum likelihood estimates (MLEs) are analogous to those computed under traditional log-linear models, and the goodness-of-fit tests are also the same. If an overall effect is not present in any reparameterization, the properties of the MLEs are considerably different, and the Poisson and multinomial MLEs are not equivalent. In the Poisson case, if the overall effect is not present, the observed total is not always preserved by the MLE, and the likelihood ratio statistic has a form which can be expressed using the Bregman divergence but does not reduce to its Kullback–Leibler version. The asymptotic equivalence chi-squared distribution of the Pearson and likelihood ratio statistics holds, but the generality considered here requires extended proofs.

On farthest Bregman Voronoi cells

ABSTRACT Let g be a strictly convex function on an evenly convex set with nonempty interior. Assuming that g is differentiable on we consider the Bregman distance associated with g. Given a set T , whose elements are called sites, and a particular site s, the farthest g-Bregman Voronoi cell of s, denoted by , consists of all points that are farther from s than from any other site with respect to . In this paper we study farthest g-Bregman Voronoi cells; in particular, we characterize those sets that can be written as for some and some .

Accelerated Bregman proximal gradient methods for relatively smooth convex optimization

We consider the problem of minimizing the sum of two convex functions: one is differentiable and relatively smooth with respect to a reference convex function, and the other can be nondifferentiable but simple to optimize. We investigate a triangle scaling property of the Bregman distance generated by the reference convex function and present accelerated Bregman proximal gradient (ABPG) methods that attain an $$O(k^{-\gamma })$$ convergence rate, where $$\gamma \in (0,2]$$ is the triangle scaling exponent (TSE) of the Bregman distance. For the Euclidean distance, we have $$\gamma =2$$ and recover the convergence rate of Nesterov’s accelerated gradient methods. For non-Euclidean Bregman distances, the TSE can be much smaller (say $$\gamma \le 1$$ ), but we show that a relaxed definition of intrinsic TSE is always equal to 2. We exploit the intrinsic TSE to develop adaptive ABPG methods that converge much faster in practice. Although theoretical guarantees on a fast convergence rate seem to be out of reach in general, our methods obtain empirical $$O(k^{-2})$$ rates in numerical experiments on several applications and provide posterior numerical certificates for the fast rates.

Halpern-Subgradient Extragradient Method for Solving Equilibrium and Common Fixed Point Problems in Reflexive Banach Spaces

In this paper, using the concept of Bregman distance, we introduce a new Bregman subgradient extragradient method for solving equilibrium and common fixed point problems in a real reflexive Banach space. The algorithm is designed, such that the stepsize is chosen without prior knowledge of the Lipschitz constants. We also prove a strong convergence result for the sequence that is generated by our algorithm under mild conditions. We apply our result to solving variational inequality problems, and finally, we give some numerical examples to illustrate the efficiency and accuracy of the algorithm.

On the uniform concentration bounds and large sample properties of clustering with Bregman divergences

Clustering with Bregman divergence has been used in literature to unify centroid‐based parametric clustering approaches and to allow the detection of nonspherical clusters within the data. Although empirically useful, the large sample theoretical aspects of Bregman clustering techniques remain largely unexplored. In this paper, we attempt to bridge the gap between the theory and practice of centroid‐based Bregman hard clustering by providing uniform deviation bounds on the clustering objective. Our theoretical analysis relies on the famous Vapnik–Chervonenkis (VC) theory, which, although has been extensively used in a supervised learning context, remains largely unexplored in finding empirical risks for unsupervised learning scenarios. As opposed to most of the theoretical works on clustering, our framework allows the number of features (p) to vary with the number of observations (n). The strong consistency of the sample cluster centroids, under standard assumptions in the literature, also follows as a corollary under this general framework. Furthermore, we show that the rate of convergence is at most of the order , under the standard regularity conditions.

Quartic First-Order Methods for Low-Rank Minimization

We study a general nonconvex formulation for low-rank minimization problems. We use recent results on non-Euclidean first-order methods to provide efficient and scalable algorithms. Our approach uses the geometry induced by the Bregman divergence of well-chosen kernel functions; for unconstrained problems, we introduce a novel family of Gram quartic kernels that improve numerical performance. Numerical experiments on Euclidean distance matrix completion and symmetric nonnegative matrix factorization show that our algorithms scale well and reach state-of-the-art performance when compared to specialized methods.

Regularisation, optimisation, subregularity

Regularisation theory in Banach spaces, and non-norm-squared regularisation even in finite dimensions, generally relies upon Bregman divergences to replace norm convergence. This is comparable to the extension of first-order optimisation methods to Banach spaces. Bregman divergences can, however, be somewhat suboptimal in terms of descriptiveness. Using the concept of (strong) metric subregularity, previously used to prove the fast local convergence of optimisation methods, we show norm convergence in Banach spaces and for non-norm-squared regularisation. For problems such as total variation regularised image reconstruction, the metric subregularity reduces to a geometric condition on the ground truth: flat areas in the ground truth have to compensate for the fidelity term not having second-order growth within the kernel of the forward operator. Our approach to proving such regularisation results is based on optimisation formulations of inverse problems. As a side result of the regularisation theory that we develop, we provide regularisation complexity results for optimisation methods: how many steps N δ of the algorithm do we have to take for the approximate solutions to converge as the corruption level δ ↘ 0?