Riemannian Gradient Research Articles

Euclidean and Poincaré space ensemble Xgboost

The Hyperbolic space has garnered attention for its unique properties and efficient representation of hierarchical structures. Recent studies have explored hyperbolic alternatives to hyperplane-based classifiers, such as logistic regression and support vector machines. Hyperbolic methods have even been fused into random forests by constructing data splits with horosphere, which proved effective for hyperbolic datasets. However, the existing incorporation of the horosphere leads to substantial computation time, diverting attention from its application on most datasets. Against this backdrop, we introduce an extension of Xgboost, a renowned machine learning (ML) algorithm to hyperbolic space, denoted as PXgboost. This extension involves a redefinition of the node split concept using the Riemannian gradient and Riemannian Hessian. Our findings unveil the promising performance of PXgboost compared to the algorithms in the literature through comprehensive experiments conducted on 64 datasets from the UCI ML repository and 8 datasets from WordNet by fusing both their Euclidean and hyperbolic-transformed (hyperbolic UCI) representations. Furthermore, our findings suggest that the Euclidean metric-based classifier performs well even on hyperbolic data. Building upon the above finding, we propose a space fusion classifier called, EPboost. It harmonizes data processing across various spaces and integrates probability outcomes for predictive analysis. In our comparative analysis involving 19 algorithms on the UCI dataset, our EPboost outperforms others in most cases, underscoring its efficacy and potential significance in diverse ML applications. This research marks a step forward in harnessing hyperbolic geometry for ML tasks and showcases its potential to enhance algorithmic efficacy.

Phase-only beampattern synthesis for maximizing mainlobe gain via Riemannian Newton method

Beampattern synthesis with phase-only weights is extensively utilized in phased-array systems due to the virtue of low-cost and high power efficiency. It is always a non-convex optimization problem having constant modulus constraints (CMCs) and the performance significantly depends on the employed solvers. This paper discusses the phase-only beampattern synthesis for maximizing the mainlobe gain with the constraints on the mainlobe ripple and the relative sidelobe level. As CMC is equivalent to a complex circle manifold, the proposed beampattern synthesis is considered as a manifold optimization problem, and then decomposed into several subproblems under the framework of manifold alternating direction method of multipliers (ADMM). In particular, the subproblem for updating phase-only weights is efficiently tackled by a second-order Riemannian Newton method, which has faster convergence speed and no requirement on line search compared to widely-used first-order Riemannian manifold methods. The explicit convergence condition is derived and the computational complexity is also analyzed. Numerical results show that the proposed method has better performance on the mainlobe gain and sidelobe control accuracy, or less computational cost compared to several representative approaches, such as convex relaxation, standard ADMM, Riemannian gradient descent-based ADMM and Riemannian conjugate gradient methods.

Differentially private Riemannian optimization

In this paper, we study the differentially private empirical risk minimization problem where the parameter is constrained to a Riemannian manifold. We introduce a framework for performing differentially private Riemannian optimization by adding noise to the Riemannian gradient on the tangent space. The noise follows a Gaussian distribution intrinsically defined with respect to the Riemannian metric on the tangent space. We adapt the Gaussian mechanism from the Euclidean space to the tangent space compatible to such generalized Gaussian distribution. This approach presents a novel analysis as compared to directly adding noise on the manifold. We further prove privacy guarantees of the proposed differentially private Riemannian (stochastic) gradient descent using an extension of the moments accountant technique. Overall, we provide utility guarantees under geodesic (strongly) convex, general nonconvex objectives as well as under the Riemannian Polyak-Łojasiewicz condition. Empirical results illustrate the versatility and efficacy of the proposed framework in several applications.

Characterization of optimization problems that are solvable iteratively with linear convergence

In this work, we state a general conjecture on the solvability of optimization problems via algorithms with linear convergence guarantees. We make a first step towards examining its correctness by fully characterizing the problems that are solvable via Riemannian gradient descent with linear convergence.

Gradient Ricci solitons on Kropina measure spaces

Let F=α2β be a Kropina metric on a measure space (M,m) with the navigation data (h,W). In this paper, we give an equivalent characterization for (M,F,m) to be a gradient almost Ricci soliton, which implies that every gradient almost Ricci soliton has vanishing SBH-curvature. Based on this, we prove that (M,F,m) is a gradient almost Ricci soliton if and only if (M,h,f) is a Riemannian gradient almost Ricci soliton, W is a unit Killing vector field on M and f satisfies a differential equation, where f is the potential function f of the measure m (see Theorem 1.2). As an application, we construct some new shrinking, steady and expanding gradient Ricci solitons.

Energy efficiency optimization algorithm based on RIS and D2D for multi-user MIMO system

Reconfigurable Intelligent Surface (RIS) is composed of reconfigurable passive components, which can realize on-demand customization of wireless communication environment. Device-to-Device (D2D) communication is one of the essential technologies to improve the capacity and spectrum efficiency of network system. In this paper, we propose an energy efficiency optimization algorithm based on RIS and D2D assisted multi-user MIMO system. The purpose is to maximize the energy efficiency of the system by jointly optimizing subcarrier allocation, power allocation and passive beamforming at RIS. We first formulate the energy efficiency maximization problem with the constraints of different user rate, power, subcarrier allocation and phase. Since it is a non-convex problem, we use an alternating optimization (AO) algorithm to decouple it into three sub-problems. The solutions of these three sub-problems are solved by subcarrier allocation based on channel strength, SCA algorithm, Dinkelbach algorithm and Riemannian gradient algorithm respectively. Simulation results show that the proposed scheme can significantly improve the energy efficiency.

Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent

We prove, under mild conditions, the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model, both in batch gradient descent and stochastic gradient descent. We also discuss a Riemannian version of the Adam algorithm. We show numerical simulations of these algorithms on various benchmarks.

RIS-Aided MIMO Systems With Hardware Impairments: Robust Beamforming Design and Analysis

Reconfigurable intelligent surface (RIS) has been anticipated to be a novel cost-effective technology to improve the performance of future wireless systems. In this paper, we investigate a practical RIS-aided multiple-input-multiple-output (MIMO) system in the presence of transceiver hardware impairments, RIS phase noise and imperfect channel state information (CSI). Joint design of the MIMO transceiver and RIS reflection matrix to minimize the total average mean-square-error (MSE) of all data streams is particularly considered. This joint design problem is non-convex and challenging to solve due to the newly considered practical imperfections. To tackle the issue, we first analyze the total average MSE by incorporating the impacts of the above system imperfections. Then, in order to handle the tightly coupled optimization variables and non-convex NP-hard constraints, an efficient iterative algorithm based on alternating optimization (AO) framework is proposed with guaranteed convergence, where each subproblem admits a closed-form optimal solution by leveraging the majorization-minimization (MM) technique. Moreover, via exploiting the special structure of the unit-modulus constraints, we propose a modified Riemannian gradient ascent (RGA) algorithm for the discrete RIS phase shift optimization. Furthermore, the optimality of the proposed algorithm is validated under line-of-sight (LoS) channel conditions, and the irreducible MSE floor effect induced by imperfections of both hardware and CSI is also revealed in the high signal-to-noise ratio (SNR) regime. Numerical results show the superior MSE performance of our proposed algorithm over the adopted benchmark schemes, and demonstrate that increasing the number of RIS elements is not always beneficial under the above system imperfections.

Non-homogeneous Riemannian gradient equations for sum of squares of Bures–Wasserstein metric

We investigate Riemannian gradient equations on the open convex cone Pm of all m×m positive definite Hermitian matrices, for the function WA(X)=∑j=1nwjdW2(X,Aj)where dW denotes the Bures–Wasserstein metric. On the special case where the gradient of the weighted sum of squares of the Bures–Wasserstein metrics vanishes, its unique solution is known as the Wasserstein mean of A1,…,An. We discuss the existence and uniqueness of solutions for non-homogeneous Riemannian gradient equations at which vector fields are constant and congruence transformations. Furthermore, we establish some bounds for these solutions.

A Riemannian Smoothing Steepest Descent Method for Non-Lipschitz Optimization on Embedded Submanifolds of Rn

In this paper, we study the generalized subdifferentials and the Riemannian gradient subconsistency that are the basis for non-Lipschitz optimization on embedded submanifolds of [Formula: see text]. We then propose a Riemannian smoothing steepest descent method for non-Lipschitz optimization on complete embedded submanifolds of [Formula: see text]. We prove that any accumulation point of the sequence generated by the Riemannian smoothing steepest descent method is a stationary point associated with the smoothing function employed in the method, which is necessary for the local optimality of the original non-Lipschitz problem. We also prove that any accumulation point of the sequence generated by our method that satisfies the Riemannian gradient subconsistency is a limiting stationary point of the original non-Lipschitz problem. Numerical experiments are conducted to demonstrate the advantages of Riemannian [Formula: see text] [Formula: see text] optimization over Riemannian [Formula: see text] optimization for finding sparse solutions and the effectiveness of the proposed method. Funding: C. Zhang was supported in part by the National Natural Science Foundation of China [Grant 12171027] and the Natural Science Foundation of Beijing [Grant 1202021]. X. Chen was supported in part by the Hong Kong Research Council [Grant PolyU15300219]. S. Ma was supported in part by the National Science Foundation [Grants DMS-2243650 and CCF-2308597], the UC Davis Center for Data Science and Artificial Intelligence Research Innovative Data Science Seed Funding Program, and a startup fund from Rice University.

Quantum State Assignment Flows.

This paper introduces assignment flows for density matrices as state spaces for representation and analysis of data associated with vertices of an underlying weighted graph. Determining an assignment flow by geometric integration of the defining dynamical system causes an interaction of the non-commuting states across the graph, and the assignment of a pure (rank-one) state to each vertex after convergence. Adopting the Riemannian-Bogoliubov-Kubo-Mori metric from information geometry leads to closed-form local expressions that can be computed efficiently and implemented in a fine-grained parallel manner. Restriction to the submanifold of commuting density matrices recovers the assignment flows for categorical probability distributions, which merely assign labels from a finite set to each data point. As shown for these flows in our prior work, the novel class of quantum state assignment flows can also be characterized as Riemannian gradient flows with respect to a non-local, non-convex potential after proper reparameterization and under mild conditions on the underlying weight function. This weight function generates the parameters of the layers of a neural network corresponding to and generated by each step of the geometric integration scheme. Numerical results indicate and illustrate the potential of the novel approach for data representation and analysis, including the representation of correlations of data across the graph by entanglement and tensorization.

Accelerated dynamics with dry friction via time scaling and averaging of doubly nonlinear evolution equations

In a Hilbert framework, for convex differentiable optimization, we analyze the long-time behavior of inertial dynamics with dry friction. In classical approaches based on asymptotically vanishing viscous damping (in accordance with Nesterov’s method), the results are expressed in terms of rapid convergence of the values of the function to its minimum value. On the other hand, dry friction induces convergence towards an approximate minimizer, typically the system stops at x when a given threshold ‖∇f(x)‖≤r is satisfied. We obtain rapid convergence results in this direction. In our approach, we start from a doubly nonlinear first-order evolution equation involving two potentials: one is the differentiable function f to be minimized, which acts on the state of the system via its gradient, and the other is the nonsmooth potential dry friction φ(x)=r‖x‖ which acts on the velocity vector via its sub-differential. To highlight the central role played by ∇f(x), we also deal with the dual formulation of these dynamics, which have a Riemannian gradient structure. We then rely on the general acceleration method recently developed by Attouch, Boţ and Nguyen, which consists in applying the method of time scaling and then averaging to a continuous differential equation of the first order in time. We thus obtain fast convergence results for second order time-evolution systems involving dry friction, asymptotically vanishing viscous damping, and Hessian-driven damping in the implicit form.

Tensor rank reduction via coordinate flows

Recently, there has been a growing interest in efficient numerical algorithms based on tensor networks and low-rank techniques to approximate high-dimensional functions and solutions to high-dimensional PDEs. In this paper, we propose a new tensor rank reduction method based on coordinate transformations that can greatly increase the efficiency of high-dimensional tensor approximation algorithms. The idea is simple: given a multivariate function, determine a coordinate transformation so that the function in the new coordinate system has smaller tensor rank. We restrict our analysis to linear coordinate transformations, which gives rise to a new class of functions that we refer to as tensor ridge functions. Leveraging Riemannian gradient descent on matrix manifolds we develop an algorithm that determines a quasi-optimal linear coordinate transformation for tensor rank reduction. The results we present for rank reduction via linear coordinate transformations open the possibility for generalizations to larger classes of nonlinear transformations.

Tensor train completion: Local recovery guarantees via Riemannian optimization

AbstractIn this work, we estimate the number of randomly selected elements of a tensor that with high probability guarantees local convergence of Riemannian gradient descent for tensor train completion. We derive a new bound for the orthogonal projections onto the tangent spaces based on the harmonic mean of the unfoldings' singular values and introduce a notion of core coherence for tensor trains. We also extend the results to tensor train completion with auxiliary subspace information and obtain the corresponding local convergence guarantees.

On Geometric Connections of Embedded and Quotient Geometries in Riemannian Fixed-Rank Matrix Optimization

In this paper, we propose a general procedure for establishing the geometric landscape connections of a Riemannian optimization problem under the embedded and quotient geometries. By applying the general procedure to the fixed-rank positive semidefinite (PSD) and general matrix optimization, we establish an exact Riemannian gradient connection under two geometries at every point on the manifold and sandwich inequalities between the spectra of Riemannian Hessians at Riemannian first-order stationary points (FOSPs). These results immediately imply an equivalence on the sets of Riemannian FOSPs, Riemannian second-order stationary points (SOSPs), and strict saddles of fixed-rank matrix optimization under the embedded and the quotient geometries. To the best of our knowledge, this is the first geometric landscape connection between the embedded and the quotient geometries for fixed-rank matrix optimization, and it provides a concrete example of how these two geometries are connected in Riemannian optimization. In addition, the effects of the Riemannian metric and quotient structure on the landscape connection are discussed. We also observe an algorithmic connection between two geometries with some specific Riemannian metrics in fixed-rank matrix optimization: there is an equivalence between gradient flows under two geometries with shared spectra of Riemannian Hessians. A number of novel ideas and technical ingredients—including a unified treatment for different Riemannian metrics, novel metrics for the Stiefel manifold, and new horizontal space representations under quotient geometries—are developed to obtain our results. The results in this paper deepen our understanding of geometric and algorithmic connections of Riemannian optimization under different Riemannian geometries and provide a few new theoretical insights to unanswered questions in the literature. Funding: X. Li was partially supported by National Key R&D Program of China [Grants 2020YFA0711900, 2020YFA0711901], the National Natural Science Foundation of China [Grants 12271107, 62141407], the Young Elite Scientists Sponsorship Program by CAST [Grant 2019QNRC001], the “Chenguang Program” by the Shanghai Education Development Foundation and Shanghai Municipal Education Commission [Grant 19CG02], the Shanghai Science and Technology Program [21JC1400600]. Y. Luo and A. R. Zhang were partially supported by the National Science Foundation [CAREER-2203741].

Operator-Valued Formulas for Riemannian Gradient and Hessian and Families of Tractable Metrics in Riemannian Optimization

Operator-Valued Formulas for Riemannian Gradient and Hessian and Families of Tractable Metrics in Riemannian Optimization

A Nonlocal Graph-PDE and Higher-Order Geometric Integration for Image Labeling

This paper introduces a novel nonlocal partial difference equation (G-PDE) for labeling metric data on graphs. The G-PDE is derived as a nonlocal reparametrization of the assignment flow approach that was introduced in [J. Math. Imaging Vision, 58 (2017), pp. 211–238]. Due to this parameterization, solving the G-PDE numerically is shown to be equivalent to computing the Riemannian gradient flow with respect to a nonconvex potential. We devise an entropy-regularized difference of convex (DC) functions decomposition of this potential and show that the basic geometric Euler scheme for integrating the assignment flow is equivalent to solving the G-PDE by an established DC programming scheme. Moreover, the viewpoint of geometric integration reveals a basic way to exploit higher-order information of the vector field that drives the assignment flow, in order to devise a novel accelerated DC programming scheme. A detailed convergence analysis of both numerical schemes is provided and illustrated by numerical experiments.

External Identification of a Reciprocal Lossy Multiport Circuit under Measurement Uncertainties by Riemannian Gradient Descent

The present paper deals with the external identification of a reciprocal, special passive, 2n-port network under measurement uncertainties. In the present context, the multiport model is represented by an admittance matrix and the condition that the network is ‘reciprocal special passive’ refers to the assumption that the real part of the admittance matrix is symmetric and positive-definite. The key point is to reformulate the identification problem as a matrix optimization program over the matrix manifold S+(2n)×S(2n). The optimization problem requires a least-squares criterion function designed to cope with over-determinacy due to the incoherent data pairs whose cardinality exceeds the problem’s number of degrees of freedom. The present paper also proposes a numerical solution to such an optimization problem based on the Riemannian-gradient steepest descent method. The numerical results show that the proposed method is effective as long as reasonable measurement error levels and problem sizes are being dealt with.

Rate Optimization for RIS-Aided mMTC Networks in the Finite Blocklength Regime

Reconfigurable intelligent surfaces (RISs) have become a promising candidate for the development of future mobile systems. In the context of massive machine-type communications (mMTC), a RIS can be used to support the transmission from a group of sensors to a collector node. Due to the short data packets, we focus on the design of the RIS for maximizing the weighted sum and minimum rates in the finite blocklength regime. Under the assumption of non-orthogonal multiple access, successive interference cancelation is considered as a decoding scheme to mitigate interference. Accordingly, we formulate the optimizations as non-convex problems and propose two sub-optimal solutions based on gradient ascent (GA) and sequential optimization (SO) with semi-definite relaxation (SDR). In the GA, we distinguish between Euclidean and Riemannian gradients. For the SO, we derive a concave lower bound for the throughput and maximize it sequentially applying SDR. Numerical results show that the SO can outperform the GA and that strategies relying on the optimization of the classical Shannon capacity might be inadequate for mMTC networks.

Hybrid Analog and Digital Beamforming for RIS-Assisted mmWave Communications

Reconfigurable intelligent surface (RIS) assisted millimeter wave (mmWave) communications has been envisioned as a prominent technology for future wireless networks, since it is capable of simultaneously providing abundant spectrum resources and favorable propagation environments. The small wavelength at mmWave bands also enables the widespread use of large antenna arrays, of which the hybrid beamforming structure has emerged as a cost-effective solution. In this paper, we aim to minimize the sum-mean-square-error (sum-MSE) in the RIS-assisted mmWave multiuser multiple input multiple output (MU-MIMO) system by jointly optimizing the hybrid analog-digital precoders and the RIS reflection matrix. We demonstrate that the role of RIS in assisting mmWave communications can be completely replaced by a large-scale Kronecker-structured hybrid array. Moreover, an accelerated Riemannian gradient algorithm using majorization minimization technique is proposed to tackle the unit-modulus constrained analog precoder/RIS design. Under the assumption of perfect channel state information (CSI), we firstly consider the single-user MIMO (SU-MIMO) setup and propose an effective alternating minimization (AM) procedure to characterize the system performance limit. Moreover, a two-stage scheme is developed for low-complexity implementation. This AM procedure is then extended to the general MU-MIMO scenario. In addition, we develop a novel enhanced regularized zero-forcing (ERZF) scheme for simultaneously combating strong noise in the low-SNR regime and mitigating multi-user interference (MUI) in the high-SNR regime. The optimality of our proposed algorithms is validated for some simplified practical scenarios. Numerical results illustrate that the proposed algorithms outperform existing benchmark schemes in terms of the actual complexity and performance.