Bregman Divergence Research Articles

Reverse em-problem based on Bregman divergence and its application to classical and quantum information theory

Reverse em-problem based on Bregman divergence and its application to classical and quantum information theory

Bregman Proximal Linearized ADMM for Minimizing Separable Sums Coupled by a Difference of Functions

AbstractIn this paper, we develop a splitting algorithm incorporating Bregman distances to solve a broad class of linearly constrained composite optimization problems, whose objective function is the separable sum of possibly nonconvex nonsmooth functions and a smooth function, coupled by a difference of functions. This structure encapsulates numerous significant nonconvex and nonsmooth optimization problems in the current literature including the linearly constrained difference-of-convex problems. Relying on the successive linearization and alternating direction method of multipliers (ADMM), the proposed algorithm exhibits the global subsequential convergence to a stationary point of the underlying problem. We also establish the convergence of the full sequence generated by our algorithm under the Kurdyka–Łojasiewicz property and some mild assumptions. The efficiency of the proposed algorithm is tested on a robust principal component analysis problem and a nonconvex optimal power flow problem.

On the linear convergence of additive Schwarz methods for the p-Laplacian

Abstract We consider additive Schwarz methods for boundary value problems involving the $p$-Laplacian. While existing theoretical estimates suggest a sublinear convergence rate for these methods, empirical evidence from numerical experiments demonstrates a linear convergence rate. In this paper we narrow the gap between these theoretical and empirical results by presenting a novel convergence analysis. First, we present a new convergence theory for additive Schwarz methods written in terms of a quasi-norm. This quasi-norm exhibits behaviour akin to the Bregman distance of the convex energy functional associated with the problem. Secondly, we provide a quasi-norm version of the Poincaré–Friedrichs inequality, which plays a crucial role in deriving a quasi-norm stable decomposition for a two-level domain decomposition setting. By utilizing these key elements we establish the asymptotic linear convergence of additive Schwarz methods for the $p$-Laplacian.

Strong convergence of Bregman projection algorithms for solving split feasibility problems

Bregman distance methods play a key role in solving problems in nonlinear analysis and optimization theory, since the Bregman distance is a useful substitute for the metric. The main purpose of this paper is to investigate two new iterative algorithms based on the Bregman distance and the Bregman projection for solving split feasibility problems in real Hilbert spaces. The algorithms are constructed around these methods: Byrne’s CQ method, Polyak’s gradient method, Halpern method, and hybrid projection method. The proposed methods involve inertial extrapolation terms and self-adaptive step sizes. We prove that the proposed iterations converge strongly to the Bregman projection of the initial point onto the solution set. Some numerical examples are provided to illustrate the computational effectiveness of our algorithms. The main results extend and improve the recent results related to the split feasibility problem.

Convergence of Peaceman-Rachford splitting method with Bregman distance for three-block nonconvex nonseparable optimization

Abstract It is of strong theoretical significance and application prospects to explore three-block nonconvex optimization with nonseparable structure, which are often modeled for many problems in machine learning, statistics, and image and signal processing. In this article, by combining the Bregman distance and Peaceman-Rachford splitting method, we propose a novel three-block Bregman Peaceman-Rachford splitting method (3-BPRSM). Under a general assumption, global convergence is presented via optimality conditions. Furthermore, we prove strong convergence when the augmented Lagrange function satisfies Kurdyka-Łojasiewicz property. In addition, if the association function possessing the Kurdyka-Łojasiewicz property exhibits a distinctive structure, then linear and sublinear convergence rate of 3-BPRSM can be guaranteed. Finally, a preliminary numerical experiment demonstrates the effectiveness.

On split equality monotone variational inclusion and fixed point problems in reflexive Banach spaces

In this paper, motivated by the works of Akbar and Shahrosvand [Filomat {\bf 32} (2018), no.\ 11, 3917-3932], Ogbuisi and Izuchukwu [Numer. Funct. Anal. Optim. {\bf 41} (2020), no.\ 2, 322-343], and some other related results in the literature, we introduce a Halpern iterative algorithm and employ a Bregman distance approach for approximating a solution of split equality monotone variational inclusion problem and fixed point problem of Bregman relatively nonexpansive mapping in reflexive Banach spaces. Under suitable condition, we state and prove a strong convergence result for approximating a common solution of the aforementioned problems. Furthermore, we give an application of our main result to variational inequality problems and provide some numerical examples to illustrate the convergence behavior of our result. The result presented in this paper extends and complements many related results in literature.

On α-Fractional Bregman Divergence to study α-Fractional Minty’s Lemma

In this paper fractional variational inequality problems (FVIP) and dual fractional variational inequality problems (DFVIP), Fractional minimization problems are defined with the help of fractional Caputo differential operator and Reimann-Liouville differential operator. The equivalence theorem of Caputo FVIP and fractional minimization problem is established. The equivalence between FVIP and DFVIP is studied with the help of fractional Bregman divergence to establish the fractional Minty’s lemma. Introduction: The theory of variational inequality problems is one of the best tool to solve the equilibrium and non-equilibrium problems arises in the branches of Applied Mathematics, Applied Physics, Engineering, Medical, Financial and many more. On the other hand, fractional calculus manages to study various types of dynamic problems arises in real and complex analysis It is certain for the development of fractional calculus because the fractional derivative has global correlation, which can reflect the historical process of the systematic function.With regard to the definition of fractional derivatives, many mathematicians studied fractional derivatives from different aspects and advanced different definitions for them, for example Riemann-Liouville (RL) derivative, Caputo derivative and other derivatives. Conclusions: The concept of -fractionally differential inequality problem is defined and studied its existence theorem with the help of -fractionally convex function. The concept of -fractionally Bregman divergence is developed and using it the existence of -fractionally minimization problem and -fractionally differential inequality problem are studied. Equivalence theorem in between -fractionally differential inequality problem and -fractionally dual differential inequality problem is studied to establish the Minty’s lemma in fractional calculus.

On the nature of Bregman functions

Let C⊂Rn be convex, compact, with nonempty interior and h be Legendre with domain C, continuous on C. We prove that h is Bregman if and only if it is strictly convex on C and C is a polytope. This provides insights on sequential convergence of many Bregman divergence based algorithm: abstract compatibility conditions between Bregman and Euclidean topology may equivalently be replaced by explicit conditions on h and C. This also emphasizes that a general convergence theory for these methods (beyond polyhedral domains) would require more refinements than Bregman's conditions.

Method with batching for stochastic finite-sum variational inequalities in non-Euclidean setting

Variational inequalities are a universal optimization paradigm that incorporate classical minimization and saddle point problems. Nowadays more and more tasks require to consider stochastic formulations of optimization problems. In this paper, we present an analysis of a method that gives optimal convergence estimates for monotone stochastic finite-sum variational inequalities. In contrast to the previous works, our method supports batching, does not lose the oracle complexity optimality and uses an arbitrary Bregman distance to take into account geometry of the problem. Paper provides experimental confirmation to algorithm’s effectiveness.

Generalized Fejér monotone sequences and their finitary content

We provide quantitative and abstract strong convergence results for sequences from a compact metric space satisfying a certain form of generalized Fejér monotonicity where (1) the metric can be replaced by a much more general type of function measuring distances (including, in particular, certain Bregman distances), (2) full Fejér monotonicity is relaxed to a partial variant and (3) the distance functions are allowed to vary along the iteration. For such sequences, the paper provides explicit and effective rates of metastability and even rates of convergence, the latter under a regularity assumption that generalizes the notion of metric regularity introduced by Kohlenbach, López-Acedo and Nicolae, itself an abstract generalization of many regularity notions from the literature. In the second part of the paper, we apply the abstract quantitative results established in the first part to two algorithms: one algorithm for approximating zeros of maximally monotone and maximally ρ-comonotone operators in Hilbert spaces (in the sense of Combettes and Pennanen as well as Bauschke, Moursi and Wang) that incorporates inertia terms every other term and another algorithm for approximating zeros of monotone operators in Banach spaces (in the sense of Browder) that is only Fejér monotone w.r.t. a certain Bregman distance.

Decentralized coordinated optimal dispatching of multi-area interconnected electricity-gas-heat system based on inertia Bregman ADMM

The coordinated interconnection of multi-area interconnected electricity-gas-heat systems (MIEGHS) has become a typical feature of the modern energy internet. While enhancing system operational efficiency, it also introduces complexity and uncertainty into system operation. In consideration of these factors, a decentralized coordinated robust optimization scheduling framework for MIEGHS is proposed to facilitate coordinated optimization. Firstly, a two-stage robust stochastic optimal dispatching model for a single-area integrated electricity-gas-heat system (IEGHS) is established, considering the uncertainty of wind power output and load. Secondly, a decentralized self-regulation control model for MIEGHS is proposed based on the principle of consistency constraints. Next, considering both the timeliness and convergence of the traditional alternating direction method of multipliers (ADMM) in solving problems, an improved ADMM method based on inertial Bregman distance is presented for the optimized solution of the proposed model. Finally, the effectiveness and superiority of the presented model and methods are verified by three interconnected IEGHS.

Inertial Halpern-type methods for variational inequality with application to medical image recovery

In this paper, we propose inertial Halpern-type algorithms involving a quasi-monotone operator for approximating solutions of variational inequality problems which are fixed points of quasi-nonexpansive mappings in reflexive Banach spaces. We use Bregman distance functions to enhance the efficiency of our algorithms and obtain strong convergence results, even in cases where the Lipschitz constant of the operator involved is unknown a priori. Furthermore, we illustrate the practical applicability of our methods through numerical experiments. Notably, our algorithms excel when compared to recent techniques in the literature. Of particular significance is their successful application in restoring computed tomography medical images that have been affected by motion blur and random noise. Our algorithms consistently outperform established state-of-the-art methods in all conducted experiments, showcasing their competitiveness and potential to advance variational inequality problem-solving, especially in the field of medical image recovery.

Understanding Higher-Order Interactions in Information Space.

Methods used in topological data analysis naturally capture higher-order interactions in point cloud data embedded in a metric space. This methodology was recently extended to data living in an information space, by which we mean a space measured with an information theoretical distance. One such setting is a finite collection of discrete probability distributions embedded in the probability simplex measured with the relative entropy (Kullback-Leibler divergence). More generally, one can work with a Bregman divergence parameterized by a different notion of entropy. While theoretical algorithms exist for this setup, there is a paucity of implementations for exploring and comparing geometric-topological properties of various information spaces. The interest of this work is therefore twofold. First, we propose the first robust algorithms and software for geometric and topological data analysis in information space. Perhaps surprisingly, despite working with Bregman divergences, our design reuses robust libraries for the Euclidean case. Second, using the new software, we take the first steps towards understanding the geometric-topological structure of these spaces. In particular, we compare them with the more familiar spaces equipped with the Euclidean and Fisher metrics.

Adaptive Bregman–Kaczmarz: an approach to solve linear inverse problems with independent noise exactly

We consider the block Bregman–Kaczmarz method for finite dimensional linear inverse problems. The block Bregman–Kaczmarz method uses blocks of the linear system and performs iterative steps with these blocks only. We assume a noise model that we call independent noise, i.e. each time the method performs a step for some block, one obtains a noisy sample of the respective part of the right-hand side which is contaminated with new noise that is independent of all previous steps of the method. One can view these noise models as making a fresh noisy measurement of the respective block each time it is used. In this framework, we are able to show that a well-chosen adaptive stepsize of the block Bregman–Kaczmarz method is able to converge to the exact solution of the linear inverse problem. The plain form of this adaptive stepsize relies on unknown quantities (like the Bregman distance to the solution), but we show a way how these quantities can be estimated purely from given data. We illustrate the finding in numerical experiments and confirm that these heuristic estimates lead to effective stepsizes.

Convex and Lorenz orders under balance correction in nonlife insurance pricing: Review and new developments

By exploiting massive amounts of data, machine learning techniques provide actuaries with predictors exhibiting high correlation with claim frequencies and severities. However, these predictors generally fail to achieve financial equilibrium and thus do not qualify as pure premiums. Autocalibration effectively addresses this issue since it ensures that every group of policyholders paying the same premium is on average self-financing. Balance correction has been proposed as a way to make any candidate premium autocalibrated with the added advantage that it improves out-of-sample Bregman divergence and hence predictive Tweedie deviance. This paper proves that balance correction is also beneficial in terms of concentration curves and derives conditions ensuring that the initial predictor and its balance-corrected version are ordered in Lorenz order. Finally, criteria are proposed to rank the balance-corrected versions of two competing predictors in the convex order.

The Extended Bregman Divergence and Parametric Estimation in Continuous Models

The Extended Bregman Divergence and Parametric Estimation in Continuous Models

Multivariate Time Series Change-Point Detection with a Novel Pearson-like Scaled Bregman Divergence.

Change-point detection is a challenging problem that has a number of applications across various real-world domains. The primary objective of CPD is to identify specific time points where the underlying system undergoes transitions between different states, each characterized by its distinct data distribution. Precise identification of change points in time series omics data can provide insights into the dynamic and temporal characteristics inherent to complex biological systems. Many change-point detection methods have traditionally focused on the direct estimation of data distributions. However, these approaches become unrealistic in high-dimensional data analysis. Density ratio methods have emerged as promising approaches for change-point detection since estimating density ratios is easier than directly estimating individual densities. Nevertheless, the divergence measures used in these methods may suffer from numerical instability during computation. Additionally, the most popular -relative Pearson divergence cannot measure the dissimilarity between two distributions of data but a mixture of distributions. To overcome the limitations of existing density ratio-based methods, we propose a novel approach called the Pearson-like scaled-Bregman divergence-based (PLsBD) density ratio estimation method for change-point detection. Our theoretical studies derive an analytical expression for the Pearson-like scaled Bregman divergence using a mixture measure. We integrate the PLsBD with a kernel regression model and apply a random sampling strategy to identify change points in both synthetic data and real-world high-dimensional genomics data of Drosophila. Our PLsBD method demonstrates superior performance compared to many other change-point detection methods.

On the Bregman-proximal iterative algorithm for the monotone inclusion problem in Banach spaces

In this paper, we focus on the solution of a class of monotone inclusion problems in reflexive Banach spaces. To reflect the geometry of the space and the operator, a more general proximal point iterative algorithm with Bregman divergence is proposed and some strong convergence results for the proposed scheme under standard assumptions are obtained. Meanwhile, the convex optimization problems and the critical point problems are studied in the applications, and the recovery of the sparse signal is simulated in the numerical experiments.

Alternating block linearized Bregman iterations for regularized nonnegative matrix factorization

In this paper, we propose an alternating block variant of the linearized Bregman iterations for a class of regularized nonnegative matrix factorization (NMF) problems. The proposed method exploits the block structure of NMF, utilizes the smooth adaptable property of the loss function based on the Bregman distance, and at the same time follows the iterative regularization idea of the linearized Bregman iterations method. Theoretically, we show that the proposed method is a descent method by adjusting the involved parameters. Finally, we end with several illustrative numerical experiments.

How to Evaluate Behavioral Models

Researchers building behavioral models, such as behavioral game theorists, use experimental data to evaluate predictive models of human behavior. However, there is little agreement about which loss function should be used in evaluations, with error rate, negative log-likelihood, cross-entropy, Brier score, and squared L2 error all being common choices. We attempt to offer a principled answer to the question of which loss functions should be used for this task, formalizing axioms that we argue loss functions should satisfy. We construct a family of loss functions, which we dub ``diagonal bounded Bregman divergences'', that satisfy all of these axioms. These rule out many loss functions used in practice, but notably include squared L2 error; we thus recommend its use for evaluating behavioral models.