Bregman Divergence Research Articles

Total Bregman divergence-driven possibilistic fuzzy clustering with kernel metric and local information for grayscale image segmentation

Kernel possibilistic fuzzy C-means with local information (KWPFLICM) has important research significance of image segmentation, but it is very sensitive to high noise or outliers. To enhance the segmentation performance of the algorithm, this paper proposes a kernelized total Bregman divergence-driven possibilistic fuzzy clustering with local information (TKWPFLICM). Firstly, a polynomial kernel function is introduced to kernelize total Bregman divergence (TBD), and local neighborhood information of the pixel is used to modify it, which overcomes the shortcomings of Bregman divergence (BD) with rotation variability; Secondly, the modified kernelized TBD and possibilistic typicality are combined to further enhance the anti-noise ability of the algorithm; Finally, the modified kernelized TBD is introduced into the objective function of KWPFLICM algorithm, then a novel robust fuzzy clustering algorithm is derived by optimization theory. Experimental results show that compared with existing fuzzy clustering-related algorithms, the average SA improvement on TKWPFLICM algorithm is in the range of 0.791% to 33.237%. Therefore, TKWPFLICM algorithm has better anti-noise robustness and segmentation accuracy.

Statistical Divergences between Densities of Truncated Exponential Families with Nested Supports: Duo Bregman and Duo Jensen Divergences.

By calculating the Kullback–Leibler divergence between two probability measures belonging to different exponential families dominated by the same measure, we obtain a formula that generalizes the ordinary Fenchel–Young divergence. Inspired by this formula, we define the duo Fenchel–Young divergence and report a majorization condition on its pair of strictly convex generators, which guarantees that this divergence is always non-negative. The duo Fenchel–Young divergence is also equivalent to a duo Bregman divergence. We show how to use these duo divergences by calculating the Kullback–Leibler divergence between densities of truncated exponential families with nested supports, and report a formula for the Kullback–Leibler divergence between truncated normal distributions. Finally, we prove that the skewed Bhattacharyya distances between truncated exponential families amount to equivalent skewed duo Jensen divergences.

Bayesian Risk With Bregman Loss: A Cramér–Rao Type Bound and Linear Estimation

A general class of Bayesian lower bounds when the underlying loss function is a Bregman divergence is demonstrated. This class can be considered as an extension of the Weinstein–Weiss family of bounds for the mean squared error and relies on finding a variational characterization of Bayesian risk. This approach allows for the derivation of a version of the Cramér–Rao bound that is specific to a given Bregman divergence. This new generalization of the Cramér–Rao bound reduces to the classical one when the loss function is taken to be the Euclidean norm. In order to evaluate the effectiveness of the new lower bounds, the paper also develops upper bounds on Bayesian risk, which are based on optimal linear estimators. The effectiveness of the new bound is evaluated in the Poisson noise setting.

Distributed Randomized Gradient-Free Mirror Descent Algorithm for Constrained Optimization

This article is concerned with the multiagent optimization problem. A distributed randomized gradient-free mirror descent (DRGFMD) method is developed by introducing a randomized gradient-free oracle in the mirror descent scheme where the non-Euclidean Bregman divergence is used. The classical gradient descent method is generalized without using subgradient information of objective functions. The proposed algorithms are the first distributed non-Euclidean zeroth-order methods, which achieve an approximate <inline-formula><tex-math notation="LaTeX">$O(\frac{1}{\sqrt{T}})$</tex-math></inline-formula> <inline-formula><tex-math notation="LaTeX">$T$</tex-math></inline-formula>-rate of convergence, recovering the best known optimal rate of distributed nonsmooth constrained convex optimization. Moreover, a decentralized reciprocal weighted averaging (RWA) approximating sequence is first investigated, the convergence for RWA sequence is shown to hold over time-varying graph. Rates of convergence are comprehensively explored for the algorithm with RWA (DRGFMD-RWA). The technique on constructing the decentralized RWA sequence provides new insight in searching for minimizers in distributed algorithms.

A stochastic variance reduction algorithm with Bregman distances for structured composite problems

We develop a novel stochastic primal–dual splitting method with Bregman distances for solving structured composite problems involving infimal convolutions in non-Euclidean spaces. The sublinear convergence in expectation of the primal–dual gap is proved under mild conditions on stepsize for the general case. The linear convergence rate is obtained under additional condition like the strong convexity relative to Bregman functions.

Adaptive and multiple time-scale eligibility traces for online deep reinforcement learning

Deep reinforcement learning (DRL) is one promising approach to teaching robots to perform complex tasks. Because methods that directly reuse the stored experience data cannot follow the change of the environment in robotic problems with a time-varying environment, online DRL is required. The eligibility traces method is well known as an online learning technique for improving sample efficiency in traditional reinforcement learning with linear regressors rather than DRL. The dependency between parameters of deep neural networks would destroy the eligibility traces, which is why they are not integrated with DRL. Although replacing the gradient with the most influential one rather than accumulating the gradients as the eligibility traces can alleviate this problem, the replacing operation reduces the number of reuses of previous experiences. To address these issues, this study proposes a new eligibility traces method that can be used even in DRL while maintaining high sample efficiency. When the accumulated gradients differ from those computed using the latest parameters, the proposed method takes into account the divergence between the past and latest parameters to adaptively decay the eligibility traces. Bregman divergences between outputs computed by the past and latest parameters are exploited due to the infeasible computational cost of the divergence between the past and latest parameters. In addition, a generalized method with multiple time-scale traces is designed for the first time. This design allows for the replacement of the most influential adaptively accumulated (decayed) eligibility traces. The proposed method outperformed conventional methods in terms of learning speed and task quality by the learned policy on benchmark tasks on a dynamic robotic simulator. A real-robot demonstration confirmed the significance of online DRL as well as the adaptability of the proposed method to a changing environment.

Learning the Proximity Operator in Unfolded ADMM for Phase Retrieval

This paper considers the phase retrieval (PR) problem, which aims to reconstruct a signal from phaseless measurements such as magnitude or power spectrograms. PR is generally handled as a minimization problem involving a quadratic loss. Recent works have considered alternative discrepancy measures, such as the Bregman divergences, but it is still challenging to tailor the optimal loss for a given setting. In this paper we propose a novel strategy to automatically learn the optimal metric for PR. We unfold a recently introduced ADMM algorithm into a neural network, and we emphasize that the information about the loss used to formulate the PR problem is conveyed by the proximity operator involved in the ADMM updates. Therefore, we replace this proximity operator with trainable activation functions: learning these in a supervised setting is then equivalent to learning an optimal metric for PR. Experiments conducted with speech signals show that our approach outperforms the baseline ADMM, using a light and interpretable neural architecture.

Burkholder inequality by Bregman divergence

Burkholder inequality by Bregman divergence

Lyapunov-Bregman functions for speed-gradient adaptive control of nonlinear time-varying systems

The paper is devoted to application of Bregman divergence to nonlinear adaptive control. Using Bregman divergence instead of quadratic function leads to a new class of Lyapunov functions in adaptive control that can be called Lyapunov-Bregman functions. An extended control algorithms design scheme is proposed and the properties of the designed control systems in presence of disturbances and time-varying parameters are studied. A special case of model reference adaptive control for nonlinear time-varying systems is studied based on the speed-gradient methodology.

Optimal solutions to variational inequalities via Bregman distances in Banach lattices

Optimal solutions to variational inequalities via Bregman distances in Banach lattices

Fast Computation of Cluster Validity Measures for Bregman Divergences and Benefits

Fast Computation of Cluster Validity Measures for Bregman Divergences and Benefits

Deep Bregman Divergence for Self-Supervised Representations Learning

Deep Bregman Divergence for Self-Supervised Representations Learning

Update of Prior Probabilities by Minimal Divergence.

The present paper investigates the update of an empirical probability distribution with the results of a new set of observations. The update reproduces the new observations and interpolates using prior information. The optimal update is obtained by minimizing either the Hellinger distance or the quadratic Bregman divergence. The results obtained by the two methods differ. Updates with information about conditional probabilities are considered as well.

Bregman primal\u2013dual first-order method and application to sparse semidefinite programming

We present a new variant of the Chambolle–Pock primal–dual algorithm with Bregman distances, analyze its convergence, and apply it to the centering problem in sparse semidefinite programming. The novelty in the method is a line search procedure for selecting suitable step sizes. The line search obviates the need for estimating the norm of the constraint matrix and the strong convexity constant of the Bregman kernel. As an application, we discuss the centering problem in large-scale semidefinite programming with sparse coefficient matrices. The logarithmic barrier function for the cone of positive semidefinite completable sparse matrices is used as the distance-generating kernel. For this distance, the complexity of evaluating the Bregman proximal operator is shown to be roughly proportional to the cost of a sparse Cholesky factorization. This is much cheaper than the standard proximal operator with Euclidean distances, which requires an eigenvalue decomposition.

Sparse sampling Kaczmarz–Motzkin method with linear convergence

The randomized sparse Kaczmarz method was recently proposed to recover sparse solutions of linear systems. In this work, we introduce a greedy variant of the randomized sparse Kaczmarz method by employing the sampling Kaczmarz–Motzkin method and prove its linear convergence in expectation with respect to the Bregman distance in the noiseless and noisy cases. This greedy variant can be viewed as a unification of the sampling Kaczmarz–Motzkin method and the randomized sparse Kaczmarz method, and hence inherits the merits of these two methods. Numerically, we report a couple of experimental results to demonstrate its superiority.

Global convergence of model function based Bregman proximal minimization algorithms

Lipschitz continuity of the gradient mapping of a continuously differentiable function plays a crucial role in designing various optimization algorithms. However, many functions arising in practical applications such as low rank matrix factorization or deep neural network problems do not have a Lipschitz continuous gradient. This led to the development of a generalized notion known as the L-smad property, which is based on generalized proximity measures called Bregman distances. However, the L-smad property cannot handle nonsmooth functions, for example, simple nonsmooth functions like vert x^4-1 vert and also many practical composite problems are out of scope. We fix this issue by proposing the MAP property, which generalizes the L-smad property and is also valid for a large class of structured nonconvex nonsmooth composite problems. Based on the proposed MAP property, we propose a globally convergent algorithm called Model BPG, that unifies several existing algorithms. The convergence analysis is based on a new Lyapunov function. We also numerically illustrate the superior performance of Model BPG on standard phase retrieval problems and Poisson linear inverse problems, when compared to a state of the art optimization method that is valid for generic nonconvex nonsmooth optimization problems.

On Bregman-Type Distances and Their Associated Projection Mappings

We investigate convergence properties of Bregman distances induced by convex representations of maximally monotone operators. We also introduce and study the projection mappings associated with such distances.

Transport information Bregman divergences

We study Bregman divergences in probability density space embedded with the $$L^2$$ –Wasserstein metric. Several properties and dualities of transport Bregman divergences are provided. In particular, we derive the transport Kullback–Leibler (KL) divergence by a Bregman divergence of negative Boltzmann–Shannon entropy in $$L^2$$ –Wasserstein space. We also derive analytical formulas and generalizations of transport KL divergence for one-dimensional probability densities and Gaussian families.

A Bregman Divergence Optimization Framework for Ranking on Data Manifold and Its New Extensions

Recently, graph-based ranking algorithms have received considerable interests in machine learning, computer vision and information retrieval communities. Ranking on data manifold (or manifold ranking, MR) is one of the representative approaches. One of the limitations of manifold ranking is its high computational complexity (O(n3), where n is the number of samples in database). In this paper, we cast the manifold ranking into a Bregman divergence optimization framework under which we transform the original MR to an equivalent optimal kernel matrix learning problem.With this new formulation, two effective and efficient extensions are proposed to enhance the ranking performance. Extensive experimental results on two real world image databases show the effectiveness of the proposed approach.

On the construction of a family of anomalous-diffusion Fokker–Planck−Kolmogorov’s equations based on the Sharma–Taneja–Mittal entropy functional

A logical scheme for constructing thermodynamics of anomalous stochastic systems based on the nonextensive two-parameter (κ, ς) -entropy of Sharma–Taneja–Mittal (SHTM) is considered. Thermodynamics within the framework (2 - q) -statistics of Tsallis was constructed, which belongs to the STM family of statistics. The approach of linear nonequilibrium thermodynamics to the construction of a family of nonlinear equations of Fokker−Planck−Kolmogorov (FPK), is used, correlated with the entropy of the STM, in which the stationary solution of the diffusion equation coincides with the corresponding generalized Gibbs distribution obtained from the extremality (κ, ς) - entropy condition of a non-additive stochastic system. Taking into account the convexity property of the Bregman divergence, it was shown that the principle of maximum equilibrium entropy is valid for (κ, ς) - systems, and also was proved the H - theorem determining the direction of the time evolution of the non-equilibrium state of the system. This result is extended also to non-equilibrium systems that evolve to a stationary state in accordance with the nonlinear FPK equation. The method of the ansatz- approach for solving non-stationary FPK equations is considered, which allows us to find the time dependence of the probability density distribution function for non-equilibrium anomalous systems. Received diffusive equations FPК can be used, in particular, at the analysis of diffusion of every possible epidemics and pandemics. The obtained diffusion equations of the FPK can be used, in particular, in the analysis of the spread of various epidemics and pandemics.