Riemannian Optimization Algorithms Research Articles

STAR-RIS aided NOMA Networks: A Signal Enhancement Algorithm

In order to overcome the reflecting-only issue of the conventional reconfigurable intelligent surface (RIS), simultaneous-transmitting-and-reflecting reconfigurable intelligent surface (STAR-RIS) stands as a potential solution for providing access services to the users located at the full space. By utilizing the omnidirectional properties of STAR-RIS, non-orthogonal multiple access (NOMA) users on both sides can be served simultaneously. By making the signal reflected by STAR-RIS coherent superposition with the direct signal, using the Riemannian optimization algorithm (ROA), the proposed signal enhancement algorithm can greatly enhance the signal power of NOMA users, reduce the outage probability, and improve the ergodic rate performance. According to the simulation results, the random phase STAR-RIS can improve the spectral efficiency of 3.6bit/s/Hz compared without STAR-RIS. After using the proposed ROA, the spectral efficiency can be extra increased by 1.8bit/s/Hz, which verifies the superiority of the proposed algorithm.

An image inpainting algorithm using exemplar matching and low-rank sparse prior

Image inpainting is a challenging problem with a wide range of applications such as object removal and old photo restoration. The methods based on low-rank sparse prior have been used for regular or nearly regular texture inpainting. However, since such inpainting results do not synthesize the original pixels, they are usually not sharp especially when the area to be recovered is large. One remedy is to use an exemplar-based method. However, it often produces false matches and cannot obtain globally consistent inpainting results. In this paper, we give a new model to promote low rankness and sparsity and solve this model with a recently proposed Riemannian optimization algorithm. Furthermore, we propose a novel two-stage algorithm by integrating the low-rank sparse model with an exemplar-based method. Numerical experiments demonstrate that the proposed low-rank sparsity-based method and the two-stage algorithm achieve encouraging results compared to state-of-the-art image completion algorithms.

Faster Riemannian Newton-type optimization by subsampling and cubic regularization

This work is on constrained large-scale non-convex optimization where the constraint set implies a manifold structure. Solving such problems is important in a multitude of fundamental machine learning tasks. Recent advances on Riemannian optimization have enabled the convenient recovery of solutions by adapting unconstrained optimization algorithms over manifolds. However, it remains challenging to scale up and meanwhile maintain stable convergence rates and handle saddle points. We propose a new second-order Riemannian optimization algorithm, aiming at improving convergence rate and reducing computational cost. It enhances the Riemannian trust-region algorithm that explores curvature information to escape saddle points through a mixture of subsampling and cubic regularization techniques. We conduct rigorous analysis to study the convergence behavior of the proposed algorithm. We also perform extensive experiments to evaluate it based on two general machine learning tasks using multiple datasets. The proposed algorithm exhibits improved computational speed, e.g., a speed improvement from 12% :text {to} :227%, and improved convergence behavior, e.g., an iteration number reduction from mathcal{O}left(maxleft(epsilon_g^{-2}epsilon_H^{-1},epsilon_H^{-3}right)right) ,text {to}: mathcal{O}left(maxleft(epsilon_g^{-2},epsilon_H^{-3}right)right), compared to a large set of state-of-the-art Riemannian optimization algorithms.

Riemannian Low-Rank Model Compression for Federated Learning With Over-the-Air Aggregation

Low-rank model compression is a widely used technique for reducing the computational load when training machine learning models. However, existing methods often rely on relaxing the low-rank constraint of the model weights using a regularized nuclear norm penalty, which requires an appropriate hyperparameter that can be difficult to determine in practice. Furthermore, existing compression techniques are not directly applicable to efficient over-the-air (OTA) aggregation in federated learning (FL) systems for distributed Internet-of-Things (IoT) scenarios. In this paper, we propose a novel manifold optimization formulation for low-rank model compression in FL that does not relax the low-rank constraint. Our optimization is conducted directly over the low-rank manifold, guaranteeing that the model is exactly low-rank. We also introduce a consensus penalty in the optimization formulation to support OTA aggregation. Based on our optimization formulation, we propose an alternating Riemannian optimization algorithm with a precoder that enables efficient OTA aggregation of low-rank local models without sacrificing training performance. Additionally, we provide convergence analysis in terms of key system parameters and conduct extensive experiments with real-world datasets to demonstrate the effectiveness of our proposed Riemannian low-rank model compression scheme compared to various state-of-the-art baselines.

Learning to Optimize on Riemannian Manifolds.

Many learning tasks are modeled as optimization problems with nonlinear constraints, such as principal component analysis and fitting a Gaussian mixture model. A popular way to solve such problems is resorting to Riemannian optimization algorithms, which yet heavily rely on both human involvement and expert knowledge about Riemannian manifolds. In this paper, we propose a Riemannian meta-optimization method to automatically learn a Riemannian optimizer. We parameterize the Riemannian optimizer by a novel recurrent network and utilize Riemannian operations to ensure that our method is faithful to the geometry of manifolds. The proposed method explores the distribution of the underlying data by minimizing the objective of updated parameters, and hence is capable of learning task-specific optimizations. We introduce a Riemannian implicit differentiation training scheme to achieve efficient training in terms of numerical stability and computational cost. Unlike conventional meta-optimization training schemes that need to differentiate through the whole optimization trajectory, our training scheme is only related to the final two optimization steps. In this way, our training scheme avoids the exploding gradient problem, and significantly reduces the computational load and memory footprint. We discuss experimental results across various constrained problems, including principal component analysis on Grassmann manifolds, face recognition, person re-identification, and texture image classification on Stiefel manifolds, clustering and similarity learning on symmetric positive definite manifolds, and few-shot learning on hyperbolic manifolds.

Tensor completion using geodesics on Segre manifolds

AbstractWe propose a Riemannian conjugate gradient algorithm for approximating incomplete tensors by canonical polyadic decompositions of low rank. Our main contribution consists of an explicit expression for an almost everywhere complete set of geodesics of the Segre manifold of rank‐1 tensors. These are leveraged in our Riemannian optimization algorithm over a geometrically convenient parametrization of rank‐ tensors to move in the direction of a tangent vector over the ‐fold product of Segre manifolds. We apply our method to movie rating predictions in a recommender system for the MovieLens dataset, and for identifying pure fluorophores via fluorescent spectroscopy with missing data. In this last application, we can recover the tensor decomposition from only of the data. In our numerical experiments, the proposed Riemannian conjugate gradient algorithm was competitive with a state‐of‐the‐art quasi‐Newton method with truncated conjugate gradient inner solves from Tensorlab in terms of accuracy and could reduce the computation time by half.

Automatic Differentiation for Riemannian Optimization on Low-Rank Matrix and Tensor-Train Manifolds

In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. Since matrices and tensors of fixed rank form smooth Riemannian manifolds, one of the popular tools for finding the low-rank approximations is to use the Riemannian optimization. Nevertheless, efficient implementation of Riemannian gradients and Hessians, required in Riemannian optimization algorithms, can be a nontrivial task in practice. Moreover, in some cases, analytic formulas are not even available. In this paper, we build upon automatic differentiation and propose a method that, given an implementation of the function to be minimized, efficiently computes Riemannian gradients and matrix-by-vector products between approximate Riemannian Hessian and a given vector.

An Efficient Riemannian Gradient Based Algorithm for Max-Cut Problems

The max-cut problem addresses the problem of finding a cut for a graph that splits the graph into two subsets of vertices so that the number of edges between these two subsets is as large as possible. However, this problem is NP-Hard, which may be solved by suboptimal algorithms. In this paper, we propose a fast and accurate Riemannian optimization algorithm for solving the max-cut problem. To do so, we develop a gradient descent algorithm and prove its convergence. Our simulation results show that the proposed method is extremely efficient on some already-investigated graphs. Specifically, our method is on average 50 times faster than the best well-known techniques with slightly losing the performance, which is on average 0.9729 of the max-cut value of the others.

Optimization on flag manifolds

A flag is a sequence of nested subspaces. Flags are ubiquitous in numerical analysis, arising in finite elements, multigrid, spectral, and pseudospectral methods for numerical pde; they arise in the form of Krylov subspaces in matrix computations, and as multiresolution analysis in wavelets constructions. They are common in statistics too—principal component, canonical correlation, and correspondence analyses may all be viewed as methods for extracting flags from a data set. The main goal of this article is to develop the tools needed for optimizing over a set of flags, which is a smooth manifold called the flag manifold, and it contains the Grassmannian as the simplest special case. We will derive closed-form analytic expressions for various differential geometric objects required for Riemannian optimization algorithms on the flag manifold; introducing various systems of extrinsic coordinates that allow us to parameterize points, metrics, tangent spaces, geodesics, distances, parallel transports, gradients, Hessians in terms of matrices and matrix operations; and thereby permitting us to formulate steepest descent, conjugate gradient, and Newton algorithms on the flag manifold using only standard numerical linear algebra.

Coupling matrix manifolds assisted optimization for optimal transport problems

Optimal transport (OT) is a powerful tool for measuring the distance between two probability distributions. In this paper, we introduce a new manifold named as the coupling matrix manifold (CMM), where each point on this novel manifold can be regarded as a transportation plan of the optimal transport problem. We firstly explore the Riemannian geometry of CMM with the metric expressed by the Fisher information. These geometrical features can be exploited in many essential optimization methods as a framework solving all types of OT problems via incorporating numerical Riemannian optimization algorithms such as gradient descent and trust region algorithms in CMM manifold. The proposed approach is validated using several OT problems in comparison with recent state-of-the-art related works. For the classic OT problem and its entropy regularized variant, it is shown that our method is comparable with the classic algorithms such as linear programming and Sinkhorn algorithms. For other types of non-entropy regularized OT problems, our proposed method has shown superior performance to other works, whereby the geometric information of the OT feasible space was not incorporated within.

A Riemannian Optimization Algorithm for Joint Maximum Likelihood Estimation of High-Dimensional Exploratory Item Factor Analysis.

There has been regained interest in joint maximum likelihood (JML) estimation of item factor analysis (IFA) recently, primarily due to its efficiency in handling high-dimensional data and numerous latent factors. It has been established under mild assumptions that the JML estimator is consistent as both the numbers of respondents and items tend to infinity. The current work presents an efficient Riemannian optimization algorithm for JML estimation of exploratory IFA with dichotomous response data, which takes advantage of the differential geometry of the fixed-rank matrix manifold. The proposed algorithm takes substantially less time to converge than a benchmark method that alternates between gradient ascent steps for person and item parameters. The performance of the proposed algorithm in the recovery of latent dimensionality, response probabilities, item parameters, and factor scores is evaluated via simulations.

Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds.

First-order non-convex Riemannian optimization algorithms have gained recent popularity in structured machine learning problems including principal component analysis and low-rank matrix completion. The current paper presents an efficient Riemannian Stochastic Path Integrated Differential EstimatoR (R-SPIDER) algorithm to solve the finite-sum and online Riemannian non-convex minimization problems. At the core of R-SPIDER is a recursive semi-stochastic gradient estimator that can accurately estimate Riemannian gradient under not only exponential mapping and parallel transport, but also general retraction and vector transport operations. Compared with prior Riemannian algorithms, such a recursive gradient estimation mechanism endows R-SPIDER with lower computational cost in first-order oracle complexity. Specifically, for finite-sum problems with n components, R-SPIDER is proved to converge to an ϵ-approximate stationary point within [Formula: see text] stochastic gradient evaluations, beating the best-known complexity [Formula: see text]; for online optimization, R-SPIDER is shown to converge with [Formula: see text] complexity which is, to the best of our knowledge, the first non-asymptotic result for online Riemannian optimization. For the special case of gradient dominated functions, we further develop a variant of R-SPIDER with improved linear rate of convergence. Extensive experimental results demonstrate the advantage of the proposed algorithms over the state-of-the-art Riemannian non-convex optimization methods.

Generalized Low-Rank Optimization for Topological Cooperation in Ultra-Dense Networks

Network densification is a natural way to support dense mobile applications under stringent requirements, such as ultra-low latency, ultra-high data rate, and massive connecting devices. Severe interference in ultra-dense networks poses a key bottleneck. Sharing channel state information (CSI) and messages across transmitters can potentially alleviate the interferences and improve the system performance. Most existing works on interference coordination require significant CSI signaling overhead and are impractical in the ultra-dense networks. This paper investigates the topological cooperation to manage interferences in message sharing based only on the network connectivity information. In particular, we propose a generalized low-rank optimization approach in a complex field to maximize the achievable degrees of freedom (DoFs) by establishing interference alignment conditions for the topological cooperation. To tackle the challenges of poor structure and non-convex rank function, we develop the Riemannian optimization algorithms to solve a sequence of complex fixed-rank subproblems through a rank growth strategy. By exploiting the non-compact Stiefel manifold formed by the set of complex full column rank matrices, we develop the Riemannian optimization algorithms to solve the complex fixed-rank optimization problem by applying the semidefinite lifting technique and the Burer–Monteiro factorization approach. The numerical results demonstrate the computational efficiency and higher DoFs achieved by the proposed algorithms.

Blind Demixing for Low-Latency Communication

In next-generation wireless networks, low-latency communication is critical to support emerging diversified applications, e.g., tactile Internet and virtual reality. In this paper, a novel blind demixing approach is developed to reduce the channel signaling overhead, thereby supporting low-latency communication. Specifically, we develop a low-rank approach to recover the original information only based on the single observed vector without any channel estimation. To address the unique challenges of multiple non-convex rank-one constraints, the quotient manifold geometry of the product of complex symmetric rank-one matrices is exploited. This is achieved by equivalently reformulating the original problem that uses complex asymmetric matrices to the one that uses Hermitian positive semidefinite matrices. We further generalize the geometric concepts of the complex product manifold via element-wise extension of the geometric concepts of the individual manifolds. The scalable Riemannian optimization algorithms, i.e., the Riemannian gradient descent algorithm and the Riemannian trust-region algorithm, are then developed to solve the blind demixing problem efficiently with low iteration complexity and low iteration cost. The statistical analysis shows that the Riemannian gradient descent with spectral initialization is guaranteed to linearly converge to the ground truth signals provided sufficient measurements. In addition, the Riemannian trust-region algorithm is provable to converge to an approximate local minimum from the arbitrary initialization point. Numerical experiments have been carried out in settings with different types of encoding matrices to demonstrate the algorithmic advantages, performance gains, and sample efficiency of the Riemannian optimization algorithms.

Flat Beam Design for Massive MIMO Systems via Riemannian Optimization

In this letter, we propose a novel algorithm to design beamformer with flat beampattern for massive multi-input-multi-output systems where the base station is equipped with large-scale uniform planar array (UPA). This flat beam can be both used for transmissions in broadcast channels and beam alignment for millimeter wave systems. Given the ideal flat beampattern, the beamforming design is formulated as the minimum mean squared error problem under the constraint of total transmit power or equal per-antenna power. A twofold alternative Riemannian optimization algorithm is proposed to solve the problem and the conventional method via multiconvex transformation is used for algorithm initialization. For UPA with massive antennas, the angle grid for beamforming should be dense enough and our proposed scheme can achieve better beampattern with even lower computational complexity compared to conventional schemes. Simulations validate the superiority of our proposed scheme.

Numerical Algorithms on the Affine Grassmannian

The affine Grassmannian is a noncompact smooth manifold that parameterizes all affine subspaces of a fixed dimension. It is a natural generalization of Euclidean space, points being zero-dimensional affine subspaces. We will realize the affine Grassmannian as a matrix manifold and extend Riemannian optimization algorithms including steepest descent, Newton method, and conjugate gradient, to real-valued functions on the affine Grassmannian. Like their counterparts for the Grassmannian, they rely only on standard numerical linear algebra and are readily computable.

ROPTLIB

Riemannian optimization is the task of finding an optimum of a real-valued function defined on a Riemannian manifold. Riemannian optimization has been a topic of much interest over the past few years due to many applications including computer vision, signal processing, and numerical linear algebra. The substantial background required to successfully design and apply Riemannian optimization algorithms is a significant impediment for many potential users. Therefore, multiple packages, such as Manopt (in Matlab) and Pymanopt (in Python), have been developed. This article describes ROPTLIB, a C++ library for Riemannian optimization. Unlike prior packages, ROPTLIB simultaneously achieves the following goals: (i) it has user-friendly interfaces in Matlab, Julia, and C++; (ii) users do not need to implement manifold- and algorithm-related objects; (iii) it provides efficient computational time due to its C++ core; (iv) it implements state-of-the-art generic Riemannian optimization algorithms, including quasi-Newton algorithms; and (v) it is based on object-oriented programming, allowing users to rapidly add new algorithms and manifolds.

Optimal projection of observations in a Bayesian setting

Optimal dimensionality reduction methods are proposed for the Bayesian inference of a Gaussian linear model with additive noise in presence of overabundant data. Three different optimal projections of the observations are proposed based on information theory: the projection that minimizes the Kullback–Leibler divergence between the posterior distributions of the original and the projected models, the one that minimizes the expected Kullback–Leibler divergence between the same distributions, and the one that maximizes the mutual information between the parameter of interest and the projected observations. The first two optimization problems are formulated as the determination of an optimal subspace and therefore the solution is computed using Riemannian optimization algorithms on the Grassmann manifold. Regarding the maximization of the mutual information, it is shown that there exists an optimal subspace that minimizes the entropy of the posterior distribution of the reduced model; a basis of the subspace can be computed as the solution to a generalized eigenvalue problem; an a priori error estimate on the mutual information is available for this particular solution; and that the dimensionality of the subspace to exactly conserve the mutual information between the input and the output of the models is less than the number of parameters to be inferred. Numerical applications to linear and nonlinear models are used to assess the efficiency of the proposed approaches, and to highlight their advantages compared to standard approaches based on the principal component analysis of the observations.

Riemannian optimization and multidisciplinary design optimization

Riemannian Optimization (RO) generalizes standard optimization methods from Euclidean spaces to Riemannian manifolds. Multidisciplinary Design Optimization (MDO) problems exist on Riemannian manifolds, and with the differential geometry framework which we have previously developed, we can now apply RO techniques to MDO. Here, we provide background theory and a literature review for RO and give the necessary formulae to implement the Steepest Descent Method (SDM), Newton’s Method (NM), and the Conjugate Gradient Method (CGM), in Riemannian form, on MDO problems. We then compare the performance of the Riemannian and Euclidean SDM, NM, and CGM algorithms on several test problems (including a satellite design problem from the MDO literature); we use a calculated step size, line search, and geodesic search in our comparisons. With the framework’s induced metric, the RO algorithms are generally not as effective as their Euclidean counterparts, and line search is consistently better than geodesic search. In our post-experimental analysis, we also show how the optimization trajectories for the Riemannian SDM and CGM relate to design coupling and thereby provide some explanation for the observed optimization behaviour. This work is only a first step in applying RO to MDO, however, and the use of quasi-Newton methods and different metrics should be explored in future research.

Guarantees of Riemannian Optimization for Low Rank Matrix Recovery

We establish theoretical recovery guarantees of a family of Riemannian optimization algorithms for low rank matrix recovery, which is about recovering an $m\times n$ rank $r$ matrix from $p < mn$ number of linear measurements. The algorithms are first interpreted as iterative hard thresholding algorithms with subspace projections. Based on this connection, we show that provided the restricted isometry constant $R_{3r}$ of the sensing operator is less than $C_\kappa /\sqrt{r}$, the Riemannian gradient descent algorithm and a restarted variant of the Riemannian conjugate gradient algorithm are guaranteed to converge linearly to the underlying rank $r$ matrix if they are initialized by one step hard thresholding. Empirical evaluation shows that the algorithms are able to recover a low rank matrix from nearly the minimum number of measurements necessary.