Riemannian Optimization Research Articles

Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds.

First-order non-convex Riemannian optimization algorithms have gained recent popularity in structured machine learning problems including principal component analysis and low-rank matrix completion. The current paper presents an efficient Riemannian Stochastic Path Integrated Differential EstimatoR (R-SPIDER) algorithm to solve the finite-sum and online Riemannian non-convex minimization problems. At the core of R-SPIDER is a recursive semi-stochastic gradient estimator that can accurately estimate Riemannian gradient under not only exponential mapping and parallel transport, but also general retraction and vector transport operations. Compared with prior Riemannian algorithms, such a recursive gradient estimation mechanism endows R-SPIDER with lower computational cost in first-order oracle complexity. Specifically, for finite-sum problems with n components, R-SPIDER is proved to converge to an ϵ-approximate stationary point within [Formula: see text] stochastic gradient evaluations, beating the best-known complexity [Formula: see text]; for online optimization, R-SPIDER is shown to converge with [Formula: see text] complexity which is, to the best of our knowledge, the first non-asymptotic result for online Riemannian optimization. For the special case of gradient dominated functions, we further develop a variant of R-SPIDER with improved linear rate of convergence. Extensive experimental results demonstrate the advantage of the proposed algorithms over the state-of-the-art Riemannian non-convex optimization methods.

Low-rank Riemannian eigensolver for high-dimensional Hamiltonians

Such problems as computation of spectra of spin chains and vibrational spectra of molecules can be written as high-dimensional eigenvalue problems, i.e., when the eigenvector can be naturally represented as a multidimensional tensor. Tensor methods have proven to be an efficient tool for the approximation of solutions of high-dimensional eigenvalue problems, however, their performance deteriorates quickly when the number of eigenstates to be computed increases. We address this issue by designing a new algorithm motivated by the ideas of Riemannian optimization (optimization on smooth manifolds) for the approximation of multiple eigenstates in the tensor-train format, which is also known as matrix product state representation. The proposed algorithm is implemented in TensorFlow, which allows for both CPU and GPU parallelization.

Computationally Efficient Data Detection Algorithm for Massive MU-MIMO Systems Using PSK Modulations

This letter considers the data detection problem in symmetric massive multiuser multiple-input-multiple-output (MU-MIMO) uplink systems, in which the number of base station (BS) antennas is equal to the number of single-antenna users. However, under uplink-heavy traffic, the conventional linear minimum mean-square-error (LMMSE)-based detectors suffer significant performance loss and thus cannot be applied. Although a detector based on Riemannian manifold optimization (RMO) provides excellent bit error rate (BER) performance that remarkably outperforms the LMMSE detector, it incurs high time complexity. Moreover, the computational complexity remains high. Thus, a simple yet computationally efficient detector based on the projected gradient descent algorithm is proposed in this letter to reduce the computational time and complexity of the detection algorithm while achieving the same BER performance as the RMO-based detector. Simulation results demonstrate that the proposed algorithm nearly yields the same performance as the RMO at low computational complexity, but the run-time complexity is considerably reduced.

Generalized Low-Rank Optimization for Topological Cooperation in Ultra-Dense Networks

Network densification is a natural way to support dense mobile applications under stringent requirements, such as ultra-low latency, ultra-high data rate, and massive connecting devices. Severe interference in ultra-dense networks poses a key bottleneck. Sharing channel state information (CSI) and messages across transmitters can potentially alleviate the interferences and improve the system performance. Most existing works on interference coordination require significant CSI signaling overhead and are impractical in the ultra-dense networks. This paper investigates the topological cooperation to manage interferences in message sharing based only on the network connectivity information. In particular, we propose a generalized low-rank optimization approach in a complex field to maximize the achievable degrees of freedom (DoFs) by establishing interference alignment conditions for the topological cooperation. To tackle the challenges of poor structure and non-convex rank function, we develop the Riemannian optimization algorithms to solve a sequence of complex fixed-rank subproblems through a rank growth strategy. By exploiting the non-compact Stiefel manifold formed by the set of complex full column rank matrices, we develop the Riemannian optimization algorithms to solve the complex fixed-rank optimization problem by applying the semidefinite lifting technique and the Burer–Monteiro factorization approach. The numerical results demonstrate the computational efficiency and higher DoFs achieved by the proposed algorithms.

Sparse coding-based representation of LBP difference for 3D/4D facial expression recognition

This paper presents an effective method for automated 3D/4D facial expression recognition based on Mesh-Local Binary Pattern Difference (mesh-LBPD). In contrast to most of existing methods, the proposed mesh-LBPD is based on a unified set of geometric and appearance features of different facial regions. Indeed, multiple features are combined into a compact form using covariance matrices, namely Cov − 3D − LBP. Then, the Cov − 3D − LBP atoms are represented as sparse data combinations. To that end, a Riemannian optimization objective for dictionary learning and sparse coding is used, in order to reduce the complexity of the problem, and the representation loss is characterized via an affine invariant Riemannian metric. In order to prove the effectiveness of the proposed compact combination of geometric and appearance features, we conducted extensive experimental validations on real-world datasets. In fact, obtained results show the capability of the proposed method to significantly outperform, or achieve comparable performances with, the state-of-the-art methods.

Accelerated Inference in Markov Random Fields via Smooth Riemannian Optimization

Markov Random Fields (MRFs) are a popular model for several pattern recognition and reconstruction problems in robotics and computer vision. Inference in MRFs is intractable in general and related work resorts to approximation algorithms. Among those techniques, semidefinite programming (SDP) relaxations have been shown to provide accurate estimates while scaling poorly with the problem size and being typically slow for practical applications. Our first contribution is to design a dual ascent method to solve standard SDP relaxations that takes advantage of the geometric structure of the problem to speed up computation. This technique, named Dual Ascent Riemannian Staircase (DARS), is able to solve large problem instances in seconds. Our second contribution is to develop a second and faster approach. The backbone of this second approach is a novel SDP relaxation combined with a fast and scalable solver based on smooth Riemannian optimization. We show that this approach, named Fast Unconstrained SEmidefinite Solver (FUSES), can solve large problems in milliseconds. Contrarily to local MRF solvers, e.g., loopy belief propagation, our approaches do not require an initial guess. Moreover, we leverage recent results from optimization theory to provide per-instance sub-optimality guarantees. We demonstrate the proposed approaches in multi-class image segmentation problems. Extensive experimental evidence shows that (i) FUSES and DARS produce near-optimal solutions, attaining an objective within 0.1% of the optimum, (ii) FUSES and DARS are remarkably faster than general-purpose SDP solvers, and FUSES is more than two orders of magnitude faster than DARS while attaining similar solution quality, (iii) FUSES is faster than local search methods while being a global solver.

Simple Algorithms for Optimization on Riemannian Manifolds with Constraints

We consider optimization problems on manifolds with equality and inequality constraints. A large body of work treats constrained optimization in Euclidean spaces. In this work, we consider extensions of existing algorithms from the Euclidean case to the Riemannian case. Thus, the variable lives on a known smooth manifold and is further constrained. In doing so, we exploit the growing literature on unconstrained Riemannian optimization. For the special case where the manifold is itself described by equality constraints, one could in principle treat the whole problem as a constrained problem in a Euclidean space. The main hypothesis we test here is whether it is sometimes better to exploit the geometry of the constraints, even if only for a subset of them. Specifically, this paper extends an augmented Lagrangian method and smoothed versions of an exact penalty method to the Riemannian case, together with some fundamental convergence results. Numerical experiments indicate some gains in computational efficiency and accuracy in some regimes for minimum balanced cut, non-negative PCA and $k$-means, especially in high dimensions.

An alternative to EM for Gaussian mixture models: batch and stochastic Riemannian optimization

We consider maximum likelihood estimation for Gaussian Mixture Models (Gmm s). This task is almost invariably solved (in theory and practice) via the Expectation Maximization (EM) algorithm. EM owes its success to various factors, of which is its ability to fulfill positive definiteness constraints in closed form is of key importance. We propose an alternative to EM grounded in the Riemannian geometry of positive definite matrices, using which we cast Gmm parameter estimation as a Riemannian optimization problem. Surprisingly, such an out-of-the-box Riemannian formulation completely fails and proves much inferior to EM. This motivates us to take a closer look at the problem geometry, and derive a better formulation that is much more amenable to Riemannian optimization. We then develop Riemannian batch and stochastic gradient algorithms that outperform EM, often substantially. We provide a non-asymptotic convergence analysis for our stochastic method, which is also the first (to our knowledge) such global analysis for Riemannian stochastic gradient. Numerous empirical results are included to demonstrate the effectiveness of our methods.

Flat Beam Design for Massive MIMO Systems via Riemannian Optimization

In this letter, we propose a novel algorithm to design beamformer with flat beampattern for massive multi-input-multi-output systems where the base station is equipped with large-scale uniform planar array (UPA). This flat beam can be both used for transmissions in broadcast channels and beam alignment for millimeter wave systems. Given the ideal flat beampattern, the beamforming design is formulated as the minimum mean squared error problem under the constraint of total transmit power or equal per-antenna power. A twofold alternative Riemannian optimization algorithm is proposed to solve the problem and the conventional method via multiconvex transformation is used for algorithm initialization. For UPA with massive antennas, the angle grid for beamforming should be dense enough and our proposed scheme can achieve better beampattern with even lower computational complexity compared to conventional schemes. Simulations validate the superiority of our proposed scheme.

Blind Demixing for Low-Latency Communication

In next-generation wireless networks, low-latency communication is critical to support emerging diversified applications, e.g., tactile Internet and virtual reality. In this paper, a novel blind demixing approach is developed to reduce the channel signaling overhead, thereby supporting low-latency communication. Specifically, we develop a low-rank approach to recover the original information only based on the single observed vector without any channel estimation. To address the unique challenges of multiple non-convex rank-one constraints, the quotient manifold geometry of the product of complex symmetric rank-one matrices is exploited. This is achieved by equivalently reformulating the original problem that uses complex asymmetric matrices to the one that uses Hermitian positive semidefinite matrices. We further generalize the geometric concepts of the complex product manifold via element-wise extension of the geometric concepts of the individual manifolds. The scalable Riemannian optimization algorithms, i.e., the Riemannian gradient descent algorithm and the Riemannian trust-region algorithm, are then developed to solve the blind demixing problem efficiently with low iteration complexity and low iteration cost. The statistical analysis shows that the Riemannian gradient descent with spectral initialization is guaranteed to linearly converge to the ground truth signals provided sufficient measurements. In addition, the Riemannian trust-region algorithm is provable to converge to an approximate local minimum from the arbitrary initialization point. Numerical experiments have been carried out in settings with different types of encoding matrices to demonstrate the algorithmic advantages, performance gains, and sample efficiency of the Riemannian optimization algorithms.

Riemannian Optimal Model Reduction of Stable Linear Systems

In this paper, we develop a method for solving the problem of minimizing the $H^2$ error norm between the transfer functions of original and reduced systems on the set of stable matrices and two Euclidean spaces. That is, we develop a method for identifying the optimal reduced system from all stable linear systems. However, it is difficult to develop an algorithm for solving this problem, because the set of stable matrices is highly non-convex. To overcome this issue, we show that the problem can be transformed into a tractable Riemannian optimization on the product manifold of the set of skew-symmetric matrices, the manifold of the symmetric positive-definite matrices, and two Euclidean spaces. The stability of the reduced systems constructed using the optimal solutions to our problem is preserved. To solve the reduced problem, the Riemannian gradient and Hessian are derived and a Riemannian trust-region method is developed. The initial point in the proposed approach is selected using the output from the balanced truncation (BT) method. Numerical experiments demonstrate that our method considerably improves the results given by BT in the sense of the $H^2$ norm, and also provides reduced systems that are globally near-optimal solutions to the problem of minimizing the $H^{\infty}$ error norm. Moreover, we show that our method provides a better reduced model than BT from the viewpoint of the frequency response.

Numerical Algorithms on the Affine Grassmannian

The affine Grassmannian is a noncompact smooth manifold that parameterizes all affine subspaces of a fixed dimension. It is a natural generalization of Euclidean space, points being zero-dimensional affine subspaces. We will realize the affine Grassmannian as a matrix manifold and extend Riemannian optimization algorithms including steepest descent, Newton method, and conjugate gradient, to real-valued functions on the affine Grassmannian. Like their counterparts for the Grassmannian, they rely only on standard numerical linear algebra and are readily computable.

Impact of Overlapped AoAs on the Achievable Uplink Rate of Hybrid Beamforming for Massive MIMO mm-Wave Systems for Industrial Environments

In this paper, we develop novel precoder and combiner schemes for multi-user hybrid digital-to-analog (D-A) beamforming for uplink massive multiple-input multiple-output millimeter wave (mm-Wave) systems, where the number of radio frequency chains is much smaller than the number of antennas each transceiver is equipped with. Industry 4.0 targets to accelerate the digitalization of manufacturing processes by allocating fixed and mobile robotics which use wireless communication. Such development requires high data throughput for which the utilization of short distance wireless communication such as mm-Wave system is crucially required. Based on our measurements, the probability of mm-Wave propagation waves to be reflected and refracted from metallic surfaces are shown to be significantly high. Consequently, uplink transmissions from different users such as robotics, machines, and sensors can go through paths sharing the same physical scatters, some transmission paths of different users may have overlapped angle of arrivals (AoAs) at the base station. Under such circumstance, the correlation between the channel vectors also increases considerably, which affects the achievable uplink rate severely. Therefore, the intrinsic focus is on substantially maximizing the desired signal while reducing the system interference. The proposed analog precoders and combiners are designed by using the power iteration and the Riemannian optimization method based on Stiefel manifold algorithms, respectively. The proposed digital combiner adapts the minimum mean-square error by judiciously exploiting the effective uplink analog channel gains. Furthermore, the channel estimation is investigated through a newly designed two-step procedure. In each scattering point, there is the strongest power point which is detected and then used to improve the accuracy by employing an angular domain scheme. The impact of this paper will be the boosting of the achievable uplink rate in industrial environments.

Supervised Dimensionality Reduction on Grassmannian for Image Set Recognition.

Modeling videos and image sets by linear subspaces has achieved great success in various visual recognition tasks. However, subspaces constructed from visual data are always notoriously embedded in a high-dimensional ambient space, which limits the applicability of existing techniques. This letter explores the possibility of proposing a geometry-aware framework for constructing lower-dimensional subspaces with maximum discriminative power from high-dimensional subspaces in the supervised scenario. In particular, we make use of Riemannian geometry and optimization techniques on matrix manifolds to learn an orthogonal projection, which shows that the learning process can be formulated as an unconstrained optimization problem on a Grassmann manifold. With this natural geometry, any metric on the Grassmann manifold can theoretically be used in our model. Experimental evaluations on several data sets show that our approach results in significantly higher accuracy than other state-of-the-art algorithms.

ROPTLIB

Riemannian optimization is the task of finding an optimum of a real-valued function defined on a Riemannian manifold. Riemannian optimization has been a topic of much interest over the past few years due to many applications including computer vision, signal processing, and numerical linear algebra. The substantial background required to successfully design and apply Riemannian optimization algorithms is a significant impediment for many potential users. Therefore, multiple packages, such as Manopt (in Matlab) and Pymanopt (in Python), have been developed. This article describes ROPTLIB, a C++ library for Riemannian optimization. Unlike prior packages, ROPTLIB simultaneously achieves the following goals: (i) it has user-friendly interfaces in Matlab, Julia, and C++; (ii) users do not need to implement manifold- and algorithm-related objects; (iii) it provides efficient computational time due to its C++ core; (iv) it implements state-of-the-art generic Riemannian optimization algorithms, including quasi-Newton algorithms; and (v) it is based on object-oriented programming, allowing users to rapidly add new algorithms and manifolds.

Joint Normalization and Dimensionality Reduction on Grassmannian: A Generalized Perspective

This letter proposes a generalized framework with joint normalization that learns lower dimensional subspaces with maximum discriminative power by using Riemannian geometry. We model the similarity/dissimilarity between subspaces using various metrics defined on Grassmannian and formulate dimensionality reduction as a nonlinear constraint optimization problem considering the orthogonalization. To obtain the linear mapping, we derive the components required to perform Riemannian optimization from the original Grassmannian through an orthonormal projection. We respect the Riemannian geometry of the Grassmann manifold and search for this projection directly from one Grassmann manifold to another face-to-face without any additional transformations. In this natural geometry-aware approach, any metric on the Grassmann manifold can theoretically reside in our model. We combine five metrics with our model, and the learning process is treated as an unconstrained optimization problem on a Grassmann manifold. Experiments on several datasets demonstrate that our approach leads to a significant accuracy gain over state-of-the-art methods.

A Riemannian trust‐region method for low‐rank tensor completion

SummaryThe goal of tensor completion is to fill in missing entries of a partially known tensor (possibly including some noise) under a low‐rank constraint. This may be formulated as a least‐squares problem. The set of tensors of a given multilinear rank is known to admit a Riemannian manifold structure; thus, methods of Riemannian optimization are applicable. In our work, we derive the Riemannian Hessian of an objective function on the low‐rank tensor manifolds using the Weingarten map, a concept from differential geometry. We discuss the convergence properties of Riemannian trust‐region methods based on the exact Hessian and standard approximations, both theoretically and numerically. We compare our approach with Riemannian tensor completion methods from recent literature, both in terms of convergence behavior and computational complexity. Our examples include the completion of randomly generated data with and without noise and the recovery of multilinear data from survey statistics.

Riemannian Optimization and Approximate Joint Diagonalization for Blind Source Separation

We consider the blind source separation (BSS) problem and the closely related approximate joint diagonalization (AJD) problem of symmetric positive definite (SPD) matrices. These two problems can be reduced to an optimization problem with three key components: the criterion to minimize, the constraint on the solution, and the optimization algorithm to solve it. This paper contains two contributions that allow us to treat these issues independently. We build the first complete Riemannian optimization framework suited for BSS and AJD handling three classical constraints, and allowing to use a large panel of general optimization algorithms on manifolds. We also perform a thorough study of the AJD problem of SPD matrices from an information geometry point of view. We study AJD criteria based on several divergences of the set of SPD matrices, provide three optimization strategies to minimize them, and analyze their properties. Our numerical experiments on simulated and pseudoreal electroencephalographic data show the interest of the Riemannian optimization framework and of the different AJD criteria we consider.

Optimal projection of observations in a Bayesian setting

Optimal dimensionality reduction methods are proposed for the Bayesian inference of a Gaussian linear model with additive noise in presence of overabundant data. Three different optimal projections of the observations are proposed based on information theory: the projection that minimizes the Kullback–Leibler divergence between the posterior distributions of the original and the projected models, the one that minimizes the expected Kullback–Leibler divergence between the same distributions, and the one that maximizes the mutual information between the parameter of interest and the projected observations. The first two optimization problems are formulated as the determination of an optimal subspace and therefore the solution is computed using Riemannian optimization algorithms on the Grassmann manifold. Regarding the maximization of the mutual information, it is shown that there exists an optimal subspace that minimizes the entropy of the posterior distribution of the reduced model; a basis of the subspace can be computed as the solution to a generalized eigenvalue problem; an a priori error estimate on the mutual information is available for this particular solution; and that the dimensionality of the subspace to exactly conserve the mutual information between the input and the output of the models is less than the number of parameters to be inferred. Numerical applications to linear and nonlinear models are used to assess the efficiency of the proposed approaches, and to highlight their advantages compared to standard approaches based on the principal component analysis of the observations.

Manifold Optimization Over the Set of Doubly Stochastic Matrices: A Second-Order Geometry

Convex optimization is a well-established research area with applications in almost all fields. Over the decades, multiple approaches have been proposed to solve convex programs. The development of interior-point methods allowed solving a more general set of convex programs known as semi-definite programs and second-order cone programs. However, it has been established that these methods are excessively slow for high dimensions, i.e., they suffer from the curse of dimensionality. On the other hand, optimization algorithms on manifold have shown great ability in finding solutions to nonconvex problems in reasonable time. This paper is interested in solving a subset of convex optimization using a different approach. The main idea behind Riemannian optimization is to view the constrained optimization problem as an unconstrained one over a restricted search space. The paper introduces three manifolds to solve convex programs under particular box constraints. The manifolds, called the doubly stochastic, symmetric and the definite multinomial manifolds, generalize the simplex also known as the multinomial manifold. The proposed manifolds and algorithms are well-adapted to solving convex programs in which the variable of interest is a multidimensional probability distribution function. Theoretical analysis and simulation results testify the efficiency of the proposed method over state of the art methods. In particular, they reveal that the proposed framework outperforms conventional generic and specialized solvers, especially in high dimensions.