Riemannian Gradient Descent Research Articles

Differentially private Riemannian optimization

In this paper, we study the differentially private empirical risk minimization problem where the parameter is constrained to a Riemannian manifold. We introduce a framework for performing differentially private Riemannian optimization by adding noise to the Riemannian gradient on the tangent space. The noise follows a Gaussian distribution intrinsically defined with respect to the Riemannian metric on the tangent space. We adapt the Gaussian mechanism from the Euclidean space to the tangent space compatible to such generalized Gaussian distribution. This approach presents a novel analysis as compared to directly adding noise on the manifold. We further prove privacy guarantees of the proposed differentially private Riemannian (stochastic) gradient descent using an extension of the moments accountant technique. Overall, we provide utility guarantees under geodesic (strongly) convex, general nonconvex objectives as well as under the Riemannian Polyak-Łojasiewicz condition. Empirical results illustrate the versatility and efficacy of the proposed framework in several applications.

Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent

We prove, under mild conditions, the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model, both in batch gradient descent and stochastic gradient descent. We also discuss a Riemannian version of the Adam algorithm. We show numerical simulations of these algorithms on various benchmarks.

Tensor rank reduction via coordinate flows

Recently, there has been a growing interest in efficient numerical algorithms based on tensor networks and low-rank techniques to approximate high-dimensional functions and solutions to high-dimensional PDEs. In this paper, we propose a new tensor rank reduction method based on coordinate transformations that can greatly increase the efficiency of high-dimensional tensor approximation algorithms. The idea is simple: given a multivariate function, determine a coordinate transformation so that the function in the new coordinate system has smaller tensor rank. We restrict our analysis to linear coordinate transformations, which gives rise to a new class of functions that we refer to as tensor ridge functions. Leveraging Riemannian gradient descent on matrix manifolds we develop an algorithm that determines a quasi-optimal linear coordinate transformation for tensor rank reduction. The results we present for rank reduction via linear coordinate transformations open the possibility for generalizations to larger classes of nonlinear transformations.

Tensor train completion: Local recovery guarantees via Riemannian optimization

AbstractIn this work, we estimate the number of randomly selected elements of a tensor that with high probability guarantees local convergence of Riemannian gradient descent for tensor train completion. We derive a new bound for the orthogonal projections onto the tangent spaces based on the harmonic mean of the unfoldings' singular values and introduce a notion of core coherence for tensor trains. We also extend the results to tensor train completion with auxiliary subspace information and obtain the corresponding local convergence guarantees.

External Identification of a Reciprocal Lossy Multiport Circuit under Measurement Uncertainties by Riemannian Gradient Descent

The present paper deals with the external identification of a reciprocal, special passive, 2n-port network under measurement uncertainties. In the present context, the multiport model is represented by an admittance matrix and the condition that the network is ‘reciprocal special passive’ refers to the assumption that the real part of the admittance matrix is symmetric and positive-definite. The key point is to reformulate the identification problem as a matrix optimization program over the matrix manifold S+(2n)×S(2n). The optimization problem requires a least-squares criterion function designed to cope with over-determinacy due to the incoherent data pairs whose cardinality exceeds the problem’s number of degrees of freedom. The present paper also proposes a numerical solution to such an optimization problem based on the Riemannian-gradient steepest descent method. The numerical results show that the proposed method is effective as long as reasonable measurement error levels and problem sizes are being dealt with.

Poincaré Fréchet mean

Generalizing the Fréchet mean from the Euclidean metric is not able to properly capture the geometric characteristics of many non-trivial operations, such as the non-dot inner product and non-Euclidean gradients defined on the manifold. One effective solution is to derive its hyperbolic representations in the Poincaré or hyperboloid model. Our goodness-of-fit testing shows that the Poincaré Fréchet mean achieves much lowerχ2power than that of the hyperboloid and typical non-linear kernels with regard to parameter perturbations. However, recent advanced optimization solvers, such as Riemannian gradient descent and minimizing upper bound, may result in imprecise convergences. This paper presents an (1−ϵ)-approximation approach to search a core-set on the Poincaré model, reducing deviations of the Poincaré Fréchet mean to its optimum. A hierarchical splitting algorithm that implicitly explores the hyperbolic representations for an arbitrary manifold is then presented. Experiments show that the (1−ϵ) Poincaré Fréchet mean adopted in hierarchical splitting, achieves better representations than Euclidean, kernel, and Lorentzian Fréchet means in graph and image data.

Exploiting Low-Rank Tensor-Train Deep Neural Networks Based on Riemannian Gradient Descent With Illustrations of Speech Processing

This work focuses on designing low-complexity hybrid tensor networks by considering trade-offs between the model complexity and practical performance. Firstly, we exploit a low-rank tensor-train deep neural network (TT-DNN) to build an end-to-end deep learning pipeline, namely LR-TT-DNN. Secondly, a hybrid model combining LR-TT-DNN with a convolutional neural network (CNN), which is denoted as CNN+(LR-TT-DNN), is set up to boost the performance. Instead of randomly assigning large TT-ranks for TT-DNN, we leverage Riemannian gradient descent to determine a TT-DNN associated with small TT-ranks. Furthermore, CNN+(LR-TT-DNN) consists of convolutional layers at the bottom for feature extraction and several TT layers at the top to solve regression and classification problems. We separately assess the LR-TT-DNN and CNN+(LR-TT-DNN) models on speech enhancement and spoken command recognition tasks. Our empirical evidence demonstrates that the LR-TT-DNN and CNN+(LR-TT-DNN) models with fewer model parameters can outperform the TT-DNN and CNN+(TT-DNN) counterparts.

Learning Linearized Assignment Flows for Image Labeling

We introduce a novel algorithm for estimating optimal parameters of linearized assignment flows for image labeling. An exact formula is derived for the parameter gradient of any loss function that is constrained by the linear system of ODEs determining the linearized assignment flow. We show how to efficiently evaluate this formula using a Krylov subspace and a low-rank approximation. This enables us to perform parameter learning by Riemannian gradient descent in the parameter space, without the need to backpropagate errors or to solve an adjoint equation. Experiments demonstrate that our method performs as good as highly-tuned machine learning software using automatic differentiation. Unlike methods employing automatic differentiation, our approach yields a low-dimensional representation of internal parameters and their dynamics which helps to understand how assignment flows and more generally neural networks work and perform.

Gradient Descent Ascent for Minimax Problems on Riemannian Manifolds.

In the paper, we study a class of useful minimax problems on Riemanian manifolds and propose a class of effective Riemanian gradient-based methods to solve these minimax problems. Specifically, we propose an effective Riemannian gradient descent ascent (RGDA) algorithm for the deterministic minimax optimization. Moreover, we prove that our RGDA has a sample complexity of O(κ2ϵ-2) for finding an ϵ-stationary solution of the Geodesically-Nonconvex Strongly-Concave (GNSC) minimax problems, where κ denotes the condition number. At the same time, we present an effective Riemannian stochastic gradient descent ascent (RSGDA) algorithm for the stochastic minimax optimization, which has a sample complexity of O(κ4ϵ-4) for finding an ϵ-stationary solution. To further reduce the sample complexity, we propose an accelerated Riemannian stochastic gradient descent ascent (Acc-RSGDA) algorithm based on the momentum-based variance-reduced technique. We prove that our Acc-RSGDA algorithm achieves a lower sample complexity of ~O(κ4ϵ-3) in searching for an ϵ-stationary solution of the GNSC minimax problems. Extensive experimental results on the robust distributional optimization and robust Deep Neural Networks (DNNs) training over Stiefel manifold demonstrate efficiency of our algorithms.

Towards a median signal detector through the total Bregman divergence and its robustness analysis

A novel family of geometric signal detectors are proposed through medians of the total Bregman divergence (TBD), which are shown advantageous over the conventional methods and their mean counterparts. By interpreting the observation data as Hermitian positive-definite matrices, their mean or median play an essential role in signal detection. As is difficult to be solved analytically, we propose numerical solutions through Riemannian gradient descent algorithms or fixed-point algorithms.Beside detection performance, robustness of a detector to outliers is also of vital importance, which can often be analyzed via the influence functions. Introducing an orthogonal basis for Hermitian matrices, we are able to compute the corresponding influence functions analytically and exactly by solving a linear system, which is transformed from the governing matrix equation. Numerical simulations show that the TBD medians are more robust than their mean counterparts.

Finding geodesics joining given points

Finding a geodesic joining two given points in a complete path-connected Riemannian manifold requires much more effort than determining a geodesic from initial data. This is because it is much harder to solve boundary value problems than initial value problems. Shooting methods attempt to solve boundary value problems by solving a sequence of initial value problems, and usually need a good initial guess to succeed. The present paper finds a geodesic gamma :[0,1]rightarrow M on the Riemannian manifold M with γ(0) = x0 and γ(1) = x1 by dividing the interval [0,1] into several sub-intervals, preferably just enough to enable a good initial guess for the boundary value problem on each subinterval. Then a geodesic joining consecutive endpoints (local junctions) is found by single shooting. Our algorithm then adjusts the junctions, either (1) by minimizing the total squared norm of the differences between associated geodesic velocities using Riemannian gradient descent, or (2) by solving a nonlinear system of equations using Newton’s method. Our algorithm is compared with the known leapfrog algorithm by numerical experiments on a 2-dimensional ellipsoid Ell(2) and on a left-invariant 3-dimensional special orthogonal group SO(3). We find Newton’s method (2) converges much faster than leapfrog when more junctions are needed, and that a good initial guess can be found for (2) by starting with Riemannian gradient descent method (1).

Energy-adaptive Riemannian optimization on the Stiefel manifold

This paper addresses the numerical solution of nonlinear eigenvector problems such as the Gross–Pitaevskii and Kohn–Sham equation arising in computational physics and chemistry. These problems characterize critical points of energy minimization problems on the infinite-dimensional Stiefel manifold. To efficiently compute minimizers, we propose a novel Riemannian gradient descent method induced by an energy-adaptive metric. Quantified convergence of the methods is established under suitable assumptions on the underlying problem. A non-monotone line search and the inexact evaluation of Riemannian gradients substantially improve the overall efficiency of the method. Numerical experiments illustrate the performance of the method and demonstrates its competitiveness with well-established schemes.

Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training

We address the poor scalability of learning algorithms for orthogonal recurrent neural networks via the use of stochastic coordinate descent on the orthogonal group, leading to a cost per iteration that increases linearly with the number of recurrent states. This contrasts with the cubic dependency of typical feasible algorithms such as stochastic Riemannian gradient descent, which prohibits the use of big network architectures. Coordinate descent rotates successively two columns of the recurrent matrix. When the coordinate (i.e., indices of rotated columns) is selected uniformly at random at each iteration, we prove convergence of the algorithm under standard assumptions on the loss function, stepsize and minibatch noise. In addition, we numerically show that the Riemannian gradient has an approximately sparse structure. Leveraging this observation, we propose a variant of our proposed algorithm that relies on the Gauss-Southwell coordinate selection rule. Experiments on a benchmark recurrent neural network training problem show that the proposed approach is a very promising step towards the training of orthogonal recurrent neural networks with big architectures.

Unsupervised Learning Discriminative MIG Detectors in Nonhomogeneous Clutter

Principal component analysis (PCA) is a commonly used pattern analysis method that maps high-dimensional data into a lower-dimensional space maximizing the data variance, that results in the promotion of separability of data. Inspired by the principle of PCA, a novel type of learning discriminative matrix information geometry (MIG) detectors in the unsupervised scenario are developed, and applied to signal detection in nonhomogeneous environments. Hermitian positive-definite (HPD) matrices can be used to model the sample data, while the clutter covariance matrix is estimated by the geometric mean of a set of secondary HPD matrices. We define a projection that maps the HPD matrices in a high-dimensional manifold to a low-dimensional and more discriminative one to increase the degree of separation of HPD matrices by maximizing the data variance. Learning a mapping can be formulated as a two-step mini-max optimization problem in Riemannian manifolds, which can be solved by the Riemannian gradient descent algorithm. Three discriminative MIG detectors are illustrated with respect to different geometric measures, i.e., the Log-Euclidean metric, the Jensen–Bregman LogDet divergence and the symmetrized Kullback–Leibler divergence. Simulation results show that performance improvements of the novel MIG detectors can be achieved compared with the conventional detectors and their state-of-the-art counterparts within nonhomogeneous environments.

New Riemannian Preconditioned Algorithms for Tensor Completion via Polyadic Decomposition

We propose new Riemannian preconditioned algorithms for low-rank tensor completion via the polyadic decomposition of a tensor. These algorithms exploit a non-Euclidean metric on the product space of the factor matrices of the low-rank tensor in the polyadic decomposition form. This new metric is designed using an approximation of the diagonal blocks of the Hessian of the tensor completion cost function, thus has a preconditioning effect on these algorithms. We prove that the proposed Riemannian gradient descent algorithm globally converges to a stationary point of the tensor completion problem, with convergence rate estimates using the $\L{}$ojasiewicz property. Numerical results on synthetic and real-world data suggest that the proposed algorithms are more efficient in memory and time compared to state-of-the-art algorithms. Moreover, the proposed algorithms display a greater tolerance for overestimated rank parameters in terms of the tensor recovery performance, thus enable a flexible choice of the rank parameter.

Generalized Low-Rank Plus Sparse Tensor Estimation by Fast Riemannian Optimization

We investigate a generalized framework to estimate a latent low-rank plus sparse tensor, where the low-rank tensor often captures the multi-way principal components and the sparse tensor accounts for potential model mis-specifications or heterogeneous signals that are unexplainable by the low-rank part. The framework flexibly covers both linear and generalized linear models, and can easily handle continuous or categorical variables. We propose a fast algorithm by integrating the Riemannian gradient descent and a novel gradient pruning procedure. Under suitable conditions, the algorithm converges linearly and can simultaneously estimate both the low-rank and sparse tensors. The statistical error bounds of final estimates are established in terms of the gradient of loss function. The error bounds are generally sharp under specific statistical models, for example, the sub-Gaussian robust PCA and Bernoulli tensor model. Moreover, our method achieves nontrivial error bounds for heavy-tailed tensor PCA whenever the noise has a finite 2 + ε moment. We apply our method to analyze the international trade flow dataset and the statistician hypergraph coauthorship network, both yielding new and interesting findings. Supplementary materials for this article are available online.

Simplification of 2D Polygonal Partitions via Point‐line Projective Duality, and Application to Urban Reconstruction

AbstractWe address the problem of simplifying two‐dimensional polygonal partitions that exhibit strong regularities. Such partitions are relevant for reconstructing urban scenes in a concise way. Preserving long linear structures spanning several partition cells motivates a point‐line projective duality approach in which points represent line intersections, and lines possibly carry multiple points. We propose a simplification algorithm that seeks a balance between the fidelity to the input partition, the enforcement of canonical relationships between lines (orthogonality or parallelism) and a low complexity output. Our methodology alternates continuous optimization by Riemannian gradient descent with combinatorial reduction, resulting in a progressive simplification scheme. Our experiments show that preserving canonical relationships helps gracefully degrade partitions of urban scenes, and yields more concise and regularity‐preserving meshes than common mesh‐based simplification approaches.

Probabilistic learning vector quantization on manifold of symmetric positive definite matrices

In this paper, we develop a new classification method for manifold-valued data in the framework of probabilistic learning vector quantization. In many classification scenarios, the data can be naturally represented by symmetric positive definite matrices, which are inherently points that live on a curved Riemannian manifold. Due to the non-Euclidean geometry of Riemannian manifolds, traditional Euclidean machine learning algorithms yield poor results on such data. In this paper, we generalize the probabilistic learning vector quantization algorithm for data points living on the manifold of symmetric positive definite matrices equipped with Riemannian natural metric (affine-invariant metric). By exploiting the induced Riemannian distance, we derive the probabilistic learning Riemannian space quantization algorithm, obtaining the learning rule through Riemannian gradient descent. Empirical investigations on synthetic data, image data , and motor imagery electroencephalogram (EEG) data demonstrate the superior performance of the proposed method.

Learning Log-Determinant Divergences for Positive Definite Matrices.

Representations in the form of Symmetric Positive Definite (SPD) matrices have been popularized in a variety of visual learning applications due to their demonstrated ability to capture rich second-order statistics of visual data. There exist several similarity measures for comparing SPD matrices with documented benefits. However, selecting an appropriate measure for a given problem remains a challenge and in most cases, is the result of a trial-and-error process. In this paper, we propose to learn similarity measures in a data-driven manner. To this end, we capitalize on the αβ-log-det divergence, which is a meta-divergence parametrized by scalars α and β, subsuming a wide family of popular information divergences on SPD matrices for distinct and discrete values of these parameters. Our key idea is to cast these parameters in a continuum and learn them from data. We systematically extend this idea to learn vector-valued parameters, thereby increasing the expressiveness of the underlying non-linear measure. We conjoin the divergence learning problem with several standard tasks in machine learning, including supervised discriminative dictionary learning and unsupervised SPD matrix clustering. We present Riemannian gradient descent schemes for optimizing our formulations efficiently, and show the usefulness of our method on eight standard computer vision tasks.

Event-triggered attitude consensus with absolute and relative attitude measurements

In this paper, we consider the event-triggered attitude consensus of multiple rigid-body systems. Two event-triggered attitude consensus protocols are designed under the absolute attitude and relative attitude measurement, respectively. For the first case, the gnomonic projection is utilized to project the attitude to the Euclidean plane almost globally. Then, a distributed attitude consensus protocol based on the projections is proposed under the event-triggered mechanism. By using the proposed protocol and event-triggered condition (ETC), the almost global attitude consensus is achieved on a positively invariant set. Next, in order to remove the requirement of the absolute attitude information, we propose an event-triggered attitude protocol with relative attitude measurements. The Riemannian gradient descent approach is utilized to design the attitude consensus protocol on a geodesically convex set of attitude configuration space. Moreover, to overcome the continuous monitoring in the event-detection, a self-triggered strategy is presented based on the event-triggered protocol only with the relative attitude measurement. Finally, simulation studies are conducted to verify the effectiveness of the proposed protocols.