Riemannian Gradient Research Articles

Continuous-domain assignment flows

Assignment flows denote a class of dynamical models for contextual data labelling (classification) on graphs. We derive a novel parametrisation of assignment flows that reveals how the underlying information geometry induces two processes for assignment regularisation and for gradually enforcing unambiguous decisions, respectively, that seamlessly interact when solving for the flow. Our result enables to characterise the dominant part of the assignment flow as a Riemannian gradient flow with respect to the underlying information geometry. We consider a continuous-domain formulation of the corresponding potential and develop a novel algorithm in terms of solving a sequence of linear elliptic partial differential equations (PDEs) subject to a simple convex constraint. Our result provides a basis for addressing learning problems by controlling such PDEs in future work.

Learning Adaptive Regularization for Image Labeling Using Geometric Assignment

We study the inverse problem of model parameter learning for pixelwise image labeling, using the linear assignment flow and training data with ground truth. This is accomplished by a Riemannian gradient flow on the manifold of parameters that determines the regularization properties of the assignment flow. Using the symplectic partitioned Runge–Kutta method for numerical integration, it is shown that deriving the sensitivity conditions of the parameter learning problem and its discretization commute. A convenient property of our approach is that learning is based on exact inference. Carefully designed experiments demonstrate the performance of our approach, the expressiveness of the mathematical model as well as its limitations, from the viewpoint of statistical learning and optimal control.

Riemannian gradient descent methods for graph-regularized matrix completion

Low-rank matrix completion is the problem of recovering the missing entries of a data matrix by using the assumption that the true matrix admits a good low-rank approximation. Much attention has been given recently to exploiting correlations between the column/row entities to improve the matrix completion quality. In this paper, we propose preconditioned gradient descent algorithms for solving the low-rank matrix completion problem with graph Laplacian-based regularizers. Experiments on synthetic data show that our approach achieves significant speedup compared to an existing method based on alternating minimization. Experimental results on real world data also show that our methods provide low-rank solutions of similar quality in comparable or less time than the state-of-the-art method.

Application of gradient descent algorithms based on geodesic distances

In this paper, the Riemannian gradient algorithm and the natural gradient algorithm are applied to solve descent direction problems on the manifold of positive definite Hermitian matrices, where the geodesic distance is considered as the cost function. The first proposed problem is control for positive definite Hermitian matrix systems whose outputs only depend on their inputs. The geodesic distance is adopted as the difference of the output matrix and the target matrix. The controller to adjust the input is obtained such that the output matrix is as close as possible to the target matrix. We show the trajectory of the control input on the manifold using the Riemannian gradient algorithm. The second application is to compute the Karcher mean of a finite set of given Toeplitz positive definite Hermitian matrices, which is defined as the minimizer of the sum of geodesic distances. To obtain more efficient iterative algorithm compared with traditional ones, a natural gradient algorithm is proposed to compute the Karcher mean. Illustrative simulations are provided to show the computational behavior of the proposed algorithms.

Manifold Optimization for Joint Design of MIMO-STAP Radars

In order to maximize the signal-to-interference-plus-noise ratio (SINR) under a constant-envelope (CE) constraint, a fast and efficient joint design of the transmit waveform and the receive filter for colocated multiple-input multiple-output (MIMO) radars is essential. Conventional joint optimization is performed using nonlinear optimization techniques such as the semidefinite relaxation (SDR) algorithm. In this letter, we propose a novel manifold-based alternating optimization (MAO) method, which reformulates the waveform optimization subproblem as an unconstrained optimization problem on a Riemannian manifold. We present the geometrical structure of the feasible region and derive the explicit expressions for the Riemannian gradient and the Riemannian Hessian, thus the reformulated optimization could be solved by using the Riemannian trust-region (RTR) algorithm. Numerical experiments demonstrate that the proposed method has faster convergence with reduced computational cost compared with conventional SDR-based algorithm in Euclidean space.

Riemannian Modified Polak--Ribière--Polyak Conjugate Gradient Order Reduced Model by Tensor Techniques

This paper presents a new Riemannian modified Polak--Ribière--Polyak conjugate gradient algorithm to construct the reduced systems of quadratic-bilinear systems. We eliminate the orthogonality and homogeneity constraints of the truncated $\mathcal{H}_{2}$ optimal model order reduction problem and turn this constrained minimization problem into an unconstrained Riemannian optimization problem on the Grassmann manifold. Due to the compactness of this manifold, the existence of the minimum solution can be guaranteed. Applying tensor techniques, the Riemannian gradient of the cost function is derived. Additionally, we design a new Riemannian MPRP conjugate gradient scheme using the differentiated retraction and the scaled vector transport. The resulting search direction always provides a descent direction. The global convergence of the proposed algorithm is established. Moreover, our algorithm is also applicable to the minimization problems of linear and bilinear systems. Finally, two numerical tests are reported to illustrate the effectiveness of the proposed algorithm.

Riemannian Geometric Optimization Methods for Joint Design of Transmit Sequence and Receive Filter on MIMO Radar

In this paper, we study the joint design of a transmit sequence and a receive filter for an airborne multiple-input multiple-output (MIMO) radar system to improve its moving target detection performance in the presence of signal-dependent interference. The optimization problem is formulated to maximize the output signal-to-noise-plus-interference ratio (SINR), subject to the waveform constant-envelope (CE) constraint. To address the challenge of this non-convex problem, we propose a novel optimization framework for solving the problem over a Riemannian manifold which is the product of complex circles and a Euclidean space. Manifold optimization views the constrained optimization problem as an unconstrained one over a restricted search space. The Riemannian gradient descent algorithms and the Riemannian trust-region algorithm are then developed to solve the reformulated problem efficiently with low iteration complexity. In addition, the proposed manifold-based algorithms provably converge to an approximate local optimum from an arbitrary initialization point. Numerical experiments demonstrate the algorithmic advantages and performance gains of the proposed algorithms.

Unsupervised Assignment Flow: Label Learning on Feature Manifolds by Spatially Regularized Geometric Assignment

This paper introduces the unsupervised assignment flow that couples the assignment flow for supervised image labeling with Riemannian gradient flows for label evolution on feature manifolds. The latter component of the approach encompasses extensions of state-of-the-art clustering approaches to manifold-valued data. Coupling label evolution with the spatially regularized assignment flow induces a sparsifying effect that enables to learn compact label dictionaries in an unsupervised manner. Our approach alleviates the requirement for supervised labeling to have proper labels at hand, because an initial set of labels can evolve and adapt to better values while being assigned to given data. The separation between feature and assignment manifolds enables the flexible application which is demonstrated for three scenarios with manifold-valued features. Experiments demonstrate a beneficial effect in both directions: adaptivity of labels improves image labeling, and steering label evolution by spatially regularized assignments leads to proper labels, because the assignment flow for supervised labeling is exactly used without any approximation for label learning.

Riemannian Structure and Flows for Smooth Geometric Image Labeling

AbstractThe image labeling problem can be described as assigning to each pixel a single element from a finite set of predefined labels. Recently, a smooth geometric approach for inferring such label assignments was proposed by following the Riemannian gradient flow of a given objective function on the so‐called assignment manifold. Due to the specific Riemannian structure, this results in a coupled replicator dynamic incorporating local spatial geometric averages of lifted data‐dependent distances. However, in this framework an approximation of the flow is necessary in order to arrive at explicit formulas. We discuss preliminary results of an alternative model where lifting and averaging is decoupled in the objective function so as to stay closer to established approaches and at the same time preserve the ingredients of the original approach. As a consequence the resulting flow is explicitly given, without the need for any approximation, while still exploiting the underlying Riemannian structure.

Characterization of solution sets of convex optimization problems in Riemannian manifolds

In this paper, a characterization of the solution sets of convex smooth optimization programmings on Riemannian manifolds, in terms of the Riemannian gradients of the cost functions, is obtained.

Energy-preserving methods on Riemannian manifolds

The energy preserving discrete gradient methods are generalized to finite-dimensional Riemannian manifolds by definition of a discrete approximation to the Riemannian gradient, a retraction, and a coordinate center function. The resulting schemes are intrinsic and do not depend on a particular choice of coordinates, nor on embedding of the manifold in a Euclidean space. Generalizations of well-known discrete gradient methods, such as the average vector field method and the Itoh--Abe method are obtained. It is shown how methods of higher order can be constructed via a collocation-like approach. Local and global error bounds are derived in terms of the Riemannian distance function and the Levi-Civita connection. Some numerical results on spin system problems are presented.

Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds.

First-order non-convex Riemannian optimization algorithms have gained recent popularity in structured machine learning problems including principal component analysis and low-rank matrix completion. The current paper presents an efficient Riemannian Stochastic Path Integrated Differential EstimatoR (R-SPIDER) algorithm to solve the finite-sum and online Riemannian non-convex minimization problems. At the core of R-SPIDER is a recursive semi-stochastic gradient estimator that can accurately estimate Riemannian gradient under not only exponential mapping and parallel transport, but also general retraction and vector transport operations. Compared with prior Riemannian algorithms, such a recursive gradient estimation mechanism endows R-SPIDER with lower computational cost in first-order oracle complexity. Specifically, for finite-sum problems with n components, R-SPIDER is proved to converge to an ϵ-approximate stationary point within [Formula: see text] stochastic gradient evaluations, beating the best-known complexity [Formula: see text]; for online optimization, R-SPIDER is shown to converge with [Formula: see text] complexity which is, to the best of our knowledge, the first non-asymptotic result for online Riemannian optimization. For the special case of gradient dominated functions, we further develop a variant of R-SPIDER with improved linear rate of convergence. Extensive experimental results demonstrate the advantage of the proposed algorithms over the state-of-the-art Riemannian non-convex optimization methods.

Blind Demixing for Low-Latency Communication

In next-generation wireless networks, low-latency communication is critical to support emerging diversified applications, e.g., tactile Internet and virtual reality. In this paper, a novel blind demixing approach is developed to reduce the channel signaling overhead, thereby supporting low-latency communication. Specifically, we develop a low-rank approach to recover the original information only based on the single observed vector without any channel estimation. To address the unique challenges of multiple non-convex rank-one constraints, the quotient manifold geometry of the product of complex symmetric rank-one matrices is exploited. This is achieved by equivalently reformulating the original problem that uses complex asymmetric matrices to the one that uses Hermitian positive semidefinite matrices. We further generalize the geometric concepts of the complex product manifold via element-wise extension of the geometric concepts of the individual manifolds. The scalable Riemannian optimization algorithms, i.e., the Riemannian gradient descent algorithm and the Riemannian trust-region algorithm, are then developed to solve the blind demixing problem efficiently with low iteration complexity and low iteration cost. The statistical analysis shows that the Riemannian gradient descent with spectral initialization is guaranteed to linearly converge to the ground truth signals provided sufficient measurements. In addition, the Riemannian trust-region algorithm is provable to converge to an approximate local minimum from the arbitrary initialization point. Numerical experiments have been carried out in settings with different types of encoding matrices to demonstrate the algorithmic advantages, performance gains, and sample efficiency of the Riemannian optimization algorithms.

Riemannian Optimal Model Reduction of Stable Linear Systems

In this paper, we develop a method for solving the problem of minimizing the $H^2$ error norm between the transfer functions of original and reduced systems on the set of stable matrices and two Euclidean spaces. That is, we develop a method for identifying the optimal reduced system from all stable linear systems. However, it is difficult to develop an algorithm for solving this problem, because the set of stable matrices is highly non-convex. To overcome this issue, we show that the problem can be transformed into a tractable Riemannian optimization on the product manifold of the set of skew-symmetric matrices, the manifold of the symmetric positive-definite matrices, and two Euclidean spaces. The stability of the reduced systems constructed using the optimal solutions to our problem is preserved. To solve the reduced problem, the Riemannian gradient and Hessian are derived and a Riemannian trust-region method is developed. The initial point in the proposed approach is selected using the output from the balanced truncation (BT) method. Numerical experiments demonstrate that our method considerably improves the results given by BT in the sense of the $H^2$ norm, and also provides reduced systems that are globally near-optimal solutions to the problem of minimizing the $H^{\infty}$ error norm. Moreover, we show that our method provides a better reduced model than BT from the viewpoint of the frequency response.

Scaling-Based Weight Normalization for Deep Neural Networks

Recent research works have highlight the invariance or the symmetry that exists in the weight space of a typical neural network and the negative effect of the symmetry on the training neural network due to the Euclidean gradient being not scaling-invariant. Although the problem of the symmetry can be solved by either defining a suitable Riemannian gradient, which is a scale-invariant or placing appropriate constraints on the weights, it will introduce very high-computation cost. In this paper, we first discuss various invariances or symmetries in the weight space, and then we propose to solve the problem via the scaling-invariance of the neural network itself, instead of the scaling-invariant updates methods. The motivation behind our method is that the optimized parameter point in the weight space may be moved from the ill-conditioning region to the flat region via a series of node-wise rescaling without changing the function represented by the neural network. Second, we proposed the scaling-based weight normalization. The proposed method can compatible with the commonly used optimization algorithms and collaborates well with batch normalization. Although our algorithm is very simple, it can accelerate the convergence speed. The additional computation cost introduced by our method is lower. Lastly, the experiments show that our proposed method can improve the performance of various networks architectures over large-scale datasets consistently. Our method outperforms the state-of-the-art methods on CIFAR-100: we obtain test errors as 17.18%.

Wasserstein Riemannian geometry of Gaussian densities

The Wasserstein distance on multivariate non-degenerate Gaussian densities is a Riemannian distance. After reviewing the properties of the distance and the metric geodesic, we present an explicit form of the Riemannian metrics on positive-definite matrices and compute its tensor form with respect to the trace inner product. The tensor is a matrix which is the solution to a Lyapunov equation. We compute the explicit formula for the Riemannian exponential, the normal coordinates charts and the Riemannian gradient. Finally, the Levi-Civita covariant derivative is computed in matrix form together with the differential equation for the parallel transport. While all computations are given in matrix form, nonetheless we discuss also the use of a special moving frame.

Some Weighted Norm Inequalities on Manifolds

Let M be a complete non-compact Riemannian manifold satisfying the volume doubling property and the Gaussian upper bounds. Denote by Δ the Laplace-Beltrami operator and by ∇ the Riemannian gradient. In this paper, the author proves the weighted reverse inequality \(\left\| {{\Delta ^{\frac{1}{2}}}f} \right\|_{L^p(w)}\leq C\left\| {|\nabla f|} \right\|_{L^p(w)}\), for some range of p determined by M and w. Moreover, a weak type estimate is proved when p = 1. Some weighted vector-valued inequalities are also established.

Riemannian optimal model reduction of linear port-Hamiltonian systems

In this paper, we describe the development of a Riemannian optimal model reduction method for linear stable port-Hamiltonian systems. This development is motivated by the fact that there remains room for improvement in existing methods. The model reduction problem is formulated as an optimization problem on the product manifold of the set of skew symmetric matrices, the manifold of the symmetric positive definite matrices, and Euclidean space. The reduced systems constructed using the optimal solutions to the problem preserve the original structure, i.e., stability, passivity, and the port-Hamiltonian form. The Riemannian gradient is derived to relate our problem to another problem in some studies, and the Hessian is also derived to solve our problem using a Riemannian trust region method. The initial point in the proposed method is chosen by using the output of the iterative rational Krylov algorithm for linear port-Hamiltonian systems (IRKA-PH). A numerical experiment illustrates that the proposed method considerably improves the results of IRKA-PH when the reduced-model dimension is small.

Global rates of convergence for nonconvex optimization on manifolds

Abstract We consider the minimization of a cost function f on a manifold $\mathcal{M}$ using Riemannian gradient descent and Riemannian trust regions (RTR). We focus on satisfying necessary optimality conditions within a tolerance ε. Specifically, we show that, under Lipschitz-type assumptions on the pullbacks of f to the tangent spaces of $\mathcal{M}$, both of these algorithms produce points with Riemannian gradient smaller than ε in $\mathcal{O}\big(1/\varepsilon ^{2}\big)$ iterations. Furthermore, RTR returns a point where also the Riemannian Hessian’s least eigenvalue is larger than −ε in $\mathcal{O} \big(1/\varepsilon ^{3}\big)$ iterations. There are no assumptions on initialization. The rates match their (sharp) unconstrained counterparts as a function of the accuracy ε (up to constants) and hence are sharp in that sense. These are the first deterministic results for global rates of convergence to approximate first- and second-order Karush-Kuhn-Tucker points on manifolds. They apply in particular for optimization constrained to compact submanifolds of ${\mathbb{R}^{n}}$, under simpler assumptions.

Guarantees of riemannian optimization for low rank matrix completion

We establish the exact recovery guarantees for a class of Riemannian optimization methods based on the embedded manifold of low rank matrices for matrix completion. Assume \begin{document}$ m $\end{document} entries of an \begin{document}$ n\times n $\end{document} rank \begin{document}$ r $\end{document} matrix are sampled independently and uniformly with replacement. We first show that with high probability the Riemannian gradient descent and conjugate gradient descent algorithms initialized by one step hard thresholding are guaranteed to converge linearly to the measured matrix provided \begin{document}$ \begin{align*} m\geq C_\kappa n^{1.5}r\log^{1.5}(n), \end{align*} $\end{document} where \begin{document}$ C_\kappa $\end{document} is a numerical constant depending on the condition number of the measured matrix. Then the sampling complexity is further improved to \begin{document}$ \begin{align*} m\geq C_\kappa nr^2\log^{2}(n) \end{align*} $\end{document} via the resampled Riemannian gradient descent initialization. The analysis of the new initialization procedure relies on an asymmetric restricted isometry property of the sampling operator and the curvature of the low rank matrix manifold. Numerical simulation shows that the algorithms are able to recover a low rank matrix from nearly the minimum number of measurements.