Adaptive metric for knowledge distillation by deep Bregman divergence.

This paper presents dual new algorithms for answering the Equilibrium Problem and Fixed Point Problems in Banach spaces. By operating the Bregman distance, we make a sweeping statement projection-based ways to unimpressed boundaries in non-Euclidean spaces, mainly in instances where conventional ideas of Lipschitz continuity or monotonicity are restrictive. The firstly algorithm services the generalized resolvent operator and Bregman projections in reflexive Banach spaces, founding strong coming together below relaxed settings. The next algorithm, planned for Hilbert spaces, integrates mistake open-mindedness machineries to confirm constancy in the attendance of computational perturbations.

This article develops a unified geometric framework linking expectation, regression, test theory, reliability, and item response theory through the concept of Bregman projection. Building on operator-theoretic and convex-analytic foundations, the framework extends the linear geometry of classical test theory (CTT) into nonlinear and information-geometric settings. Reliability and regression emerge as measures of projection efficiency-linear in Hilbert space and nonlinear under convex potentials. The exposition demonstrates that classical conditional expectation, least-squares regression, and information projections in exponential-family models share a common mathematical structure defined by Bregman divergence. By situating CTT within this broader geometric context, the article clarifies relationships between measurement, expectation, and statistical inference, providing a coherent foundation for nonlinear measurement and estimation in psychometrics.

This paper introduces a broad class of Mirror Descent (MD) and Generalized Exponentiated Gradient (GEG) algorithms derived from trace-form entropies defined via deformed logarithms. Leveraging these generalized entropies yields MD and GEG algorithms with improved convergence behavior, robustness against vanishing and exploding gradients, and inherent adaptability to non-Euclidean geometries through mirror maps. We establish deep connections between these methods and Amari's natural gradient, revealing a unified geometric foundation for additive, multiplicative, and natural gradient updates. Focusing on the Tsallis, Kaniadakis, Sharma-Taneja-Mittal, and Kaniadakis-Lissia-Scarfone entropy families, we show that each entropy induces a distinct Riemannian metric on the parameter space, leading to GEG algorithms that preserve the natural statistical geometry. The tunable parameters of deformed logarithms enable adaptive geometric selection, providing enhanced robustness and convergence over classical Euclidean optimization. Overall, our framework unifies key first-order MD optimization methods under a single information-geometric perspective based on generalized Bregman divergences, where the choice of entropy determines the underlying metric and dual geometric structure.

Abstract We propose and analyze a class of linear regularization methods for inverse problems in Banach spaces based on the profound application of abstract functional calculus (AFC). Our main novelties include error estimates, convergence rates, and converse results for the proposed approach in the deterministic and random cases. These results are established based on certain AFC with scaling-invariant properties and a suitable AFC description of real interpolation spaces, which we design and explore for the first time in the context of inverse problems. In particular, our results circumvent the use of Bregman distance and avoid using any extension of qualification and index functions, which are widely used in the Hilbertian setting. The achieved results are applicable to prominent instances, including the Tikhonov, Showalter, and Lavrentiev regularization methods. In the final part of the paper, we report on some numerical tests conforming our theoretical findings.

In this paper, we propose a new primal-dual algorithmic framework for a class of convex-concave saddle point problems frequently arising from image processing and machine learning. Our algorithmic framework updates the primal variable between the twice calculations of the dual variable, thereby appearing a symmetric iterative scheme, which is accordingly called the symmetric primal-dual algorithm (SPIDA). It is noteworthy that the subproblems of our SPIDA are equipped with Bregman proximal regularization terms, which make SPIDA versatile in the sense that it enjoys an algorithmic framework to understand the iterative schemes of some existing algorithms, such as the classical augmented Lagrangian method (ALM), linearized ALM, and Jacobian splitting algorithms for linearly constrained optimization problems. Besides, our algorithmic framework allows us to derive some customized versions so that SPIDA works as efficiently as possible for structured optimization problems. Theoretically, under some mild conditions, we prove the global convergence of SPIDA and estimate the linear convergence rate under a generalized error bound condition defined by Bregman distance. Finally, a series of numerical experiments on the basis pursuit, robust principal component analysis, and image restoration demonstrate that our SPIDA works well on synthetic and real-world datasets.

In this paper, an inertial shrinking projection algorithm with adaptive step size is introduced for solving monotone inclusion problems in reflexive Banach spaces. Properties of the Bregman distance and inertial coefficients are utilized to construct a specific family of shrinking projection sets that reflect the geometric characteristics of the inertial term. Strong convergence results for the proposed scheme are obtained under mild assumptions. Several numerical experiments are conducted to demonstrate the effectiveness of the algorithm.

In this paper, we consider the online proximal mirror descent for solving the time-varying composite optimization problems. For various applications, the algorithm naturally involves the errors in the gradient and proximal operator. We obtain sharp estimates on the dynamic regret of the algorithm when the regular part of the cost is convex and smooth. If the Bregman distance is given by the Euclidean distance, our result also improves the previous work in two ways: (i) We establish a sharper regret bound compared to the previous work in the sense that our estimate does not involve O ( T ) term appearing in that work. (ii) We also obtain the result when the domain is the whole space R n , whereas the previous work was obtained only for bounded domains. We also provide numerical tests for problems involving the errors in the gradient and proximal operator.

Abstract We consider the minimization of $$\ell _0$$ ℓ 0 -regularized criteria involving non-quadratic data terms such as the Kullback-Leibler divergence and the logistic regression, possibly combined with an $$\ell _2$$ ℓ 2 regularization. We first prove the existence of global minimizers for such problems and characterize their local minimizers. Then, we propose a new class of continuous relaxations of the $$\ell _0$$ ℓ 0 pseudo-norm, termed as $$\ell _0$$ ℓ 0 Bregman Relaxations (B-rex). They are defined in terms of suitable Bregman distances and lead to exact continuous relaxations of the original $$\ell _0$$ ℓ 0 -regularized problem in the sense that they do not alter its set of global minimizers and reduce its non-convexity by eliminating certain local minimizers. Both features make such relaxed problems more amenable to be solved by standard non-convex optimization algorithms. In this spirit, we consider the proximal gradient algorithm and provide explicit computation of proximal points for the B-rex penalty in several cases. Finally, we report a set of numerical results illustrating the geometrical behavior of the proposed B-rex penalty for different choices of the underlying Bregman distance, its relation with convex envelopes, as well as its exact relaxation properties in 1D/2D and higher dimensions.

In this work, we propose MirrorCBO, a consensus-based optimization (CBO) method which generalizes standard CBO in the same way that mirror descent generalizes gradient descent. For this, we apply the CBO methodology to a swarm of dual particles and retain the primal particle positions by applying the inverse of the mirror map, which we parametrize as the subdifferential of a strongly convex function [Formula: see text]. In this way, we combine the advantages of a derivative-free non-convex optimization algorithm with those of mirror descent. As a special case, the method extends CBO to optimization problems with convex constraints. Assuming bounds on the Bregman distance associated to [Formula: see text], we provide asymptotic convergence results for MirrorCBO with explicit exponential rate. Another key contribution is an exploratory numerical study of this new algorithm across different application settings, focusing on (i) sparsity-inducing optimization, and (ii) constrained optimization, demonstrating the competitive performance of MirrorCBO. We observe empirically that the method can also be used for optimization on (non-convex) submanifolds of Euclidean space, can be adapted to mirrored versions of other recent CBO variants, and that it inherits from mirror descent the capability to select desirable minimizers, like sparse ones. We also include an overview of recent CBO approaches for constrained optimization and compare their performance to MirrorCBO.

Kernelized weighted local information based picture fuzzy clustering with multivariate coefficient of variation and modified total Bregman divergence measure for brain MRI image segmentation.

Abstract Integrated optical phased arrays (OPAs) have shown promise in applications such as optical wireless communications and light detection and ranging (LiDAR). Existing beam‐forming methods with such OPAs often require iterations of phase‐tuning and far‐field measurements or need expensive resources to handle nonlinear dependence on elemental phases, which are inefficient and costly for generating a large set of diverse beam patterns. Here, a generic complex far‐field pattern‐forming method, the Arbitrary Precise Pattern former (APP‐former), is proposed for OPAs without iterative measurements. To tackle the complicated phase dependence, the problem is converted with auxiliary variables while ensuring the meaningful characteristics of phase variables. By introducing Bregman divergence for linearization, a closed‐form solution is obtained for each optimization step, which avoids nested double‐iterations and enhances convergence saliently. Four representative categories of patterns are demonstrated in experiments without iterative rephasing and measurements. The potential application of multi‐beam is illustrated in a multi‐user optical wireless communication network, with directional nulling to enable privacy‐oriented communications. Furthermore, the APP‐former can be adapted to uncover the dormant potential of the phase shifter array for power optimization. This versatile and efficient approach opens the door to on‐demand beam‐forming with integrated OPAs for a wide range of applications.

Motivated by questions arising at the intersection of information theory and geometry, we compare two dissimilarity measures between finite categorical distributions. One is the well-known Jensen–Shannon divergence, which is easy to compute and whose square root is a proper metric. The other is what we call the minmax divergence, which is harder to compute. Just like the Jensen–Shannon divergence, it arises naturally from the Kullback–Leibler divergence. The main contribution of this paper is a proof showing that the minmax divergence can be tightly approximated by the Jensen–Shannon divergence. The bounds suggest that the square root of the minmax divergence is a metric, and we prove that this is indeed true in the one-dimensional case. The general case remains open. Finally, we consider analogous questions in the context of another Bregman divergence and the corresponding Burbea–Rao (Jensen–Bregman) divergence.

In this paper, our main focus is on the Landweber–Kaczmarz iteration with the general uniform convex penalty function. Based on the convex analysis, we generalized that the penalty term can be non-smooth to include L 1 function space and total variation-like penalty function. By the use of subdifferential calculus, Bregman distance and Fréchet differentiable, we get monotony results. Furthermore, we obtain some results of the stability and convergence about the Landweber–Kaczmarz iterative algorithm under exact data and noisy data.

Sequential splitting algorithms with Bregman distance for solving equilibrium problems

Point cloud denoising is essential for improving 3D data quality, yet traditional K-means methods relying on Euclidean distance struggle with non-uniform noise. This paper proposes a K-means algorithm leveraging Total Bregman Divergence (TBD) to better model geometric structures on manifolds, enhancing robustness against noise. Specifically, TBDs—Total Logarithm, Exponential, and Inverse Divergences—are defined on symmetric positive-definite matrices, each tailored to capture distinct local geometries. Theoretical analysis demonstrates the bounded sensitivity of TBD-induced means to outliers via influence functions, while anisotropy indices quantify structural variations. Numerical experiments validate the method’s superiority over Euclidean-based approaches, showing effective noise separation and improved stability. This work bridges geometric insights with practical clustering, offering a robust framework for point cloud preprocessing in vision and robotics applications.

Bregman divergences form a class of distance-like comparison functions which plays fundamental roles in optimization, statistics, and information theory. One important property of Bregman divergences is that they generate agreement between two useful formulations of information content (in the sense of variability or non-uniformity) in weighted collections of vectors. The first of these is the Jensen gap information, which measures the difference between the mean value of a strictly convex function evaluated on a weighted set of vectors and the value of that function evaluated at the centroid of that collection. The second of these is the divergence information, which measures the mean divergence of the vectors in the collection from their centroid. In this brief note, we prove that the agreement between Jensen gap and divergence informations in fact characterizes the class of Bregman divergences; they are the only divergences that generate this agreement for arbitrary weighted sets of data vectors.

A Proximal-Type Algorithm with Bregman Distance for Solving Equilibrium Problems

This article introduces a unified approach to estimating the mutual density ratio, defined as the ratio between the joint density function and the product of the individual marginal density functions of two random vectors. It serves as a fundamental measure for quantifying the relationship between two random vectors. Our method uses the Bregman divergence to construct the objective function and leverages deep neural networks to approximate the logarithm of the mutual density ratio. We establish a non-asymptotic error bound for our estimator, achieving the optimal minimax rate of convergence under a bounded support condition. Additionally, our estimator mitigates the curse of dimensionality when the distribution is supported on a lower-dimensional manifold. We extend our results to overparameterized neural networks and the case with unbounded support. Applications of our method include conditional probability density estimation, mutual information estimation, and independence testing. Simulation studies and real data examples demonstrate the effectiveness of our approach. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

An agent acquires information dynamically until her belief about a binary state reaches an upper or lower threshold. She can choose any signal process subject to a constraint on the rate of entropy reduction. Strategies are ordered by “time risk”—the dispersion of the distribution of threshold-hitting times. We construct a strategy maximizing time risk (Greedy Exploitation) and one minimizing it (Pure Accumulation). Under either, beliefs follow a compensated Poisson process. In the former, beliefs jump to the threshold closer in Bregman divergence. In the latter, beliefs jump to the unique point with the same entropy as the current belief. (JEL D81, D82, D83)