Stiefel Manifold Research Articles

Learning to Optimize on Riemannian Manifolds.

Many learning tasks are modeled as optimization problems with nonlinear constraints, such as principal component analysis and fitting a Gaussian mixture model. A popular way to solve such problems is resorting to Riemannian optimization algorithms, which yet heavily rely on both human involvement and expert knowledge about Riemannian manifolds. In this paper, we propose a Riemannian meta-optimization method to automatically learn a Riemannian optimizer. We parameterize the Riemannian optimizer by a novel recurrent network and utilize Riemannian operations to ensure that our method is faithful to the geometry of manifolds. The proposed method explores the distribution of the underlying data by minimizing the objective of updated parameters, and hence is capable of learning task-specific optimizations. We introduce a Riemannian implicit differentiation training scheme to achieve efficient training in terms of numerical stability and computational cost. Unlike conventional meta-optimization training schemes that need to differentiate through the whole optimization trajectory, our training scheme is only related to the final two optimization steps. In this way, our training scheme avoids the exploding gradient problem, and significantly reduces the computational load and memory footprint. We discuss experimental results across various constrained problems, including principal component analysis on Grassmann manifolds, face recognition, person re-identification, and texture image classification on Stiefel manifolds, clustering and similarity learning on symmetric positive definite manifolds, and few-shot learning on hyperbolic manifolds.

LDA-MIG Detectors for Maritime Targets in Nonhomogeneous Sea Clutter

This paper deals with the problem of detecting maritime targets embedded in nonhomogeneous sea clutter, where limited number of secondary data is available due to the heterogeneity of sea clutter. A class of linear discriminant analysis (LDA)-based matrix information geometry (MIG) detectors is proposed in the supervised scenario. As customary, Hermitian positive-definite (HPD) matrices are used to model the observational sample data, and the clutter covariance matrix of received dataset is estimated as geometric mean of the secondary HPD matrices. Given a set of training HPD matrices with class labels, that are elements of a higher-dimensional HPD matrix manifold, the LDA manifold projection learns a mapping from the higher-dimensional HPD matrix manifold to a lower-dimensional one subject to maximum discrimination. In the current study, the LDA manifold projection, with the cost function maximizing between-class distance while minimizing within-class distance, is formulated as an optimization problem in the Stiefel manifold. Four robust LDA-MIG detectors corresponding to different geometric measures are proposed. Numerical results based on both simulated radar clutter with interferences and real IPIX radar data show the advantage of the proposed LDA-MIG detectors against their counterparts without using LDA as well as the state-of-art maritime target detection methods in nonhomogeneous sea clutter.

Large deviation principles induced by the Stiefel manifold, and random multidimensional projections

Large deviation principles induced by the Stiefel manifold, and random multidimensional projections

Higher order concentration on Stiefel and Grassmann manifolds

Higher order concentration on Stiefel and Grassmann manifolds

Conjugate Gradient Methods for Optimization Problems on Symplectic Stiefel Manifold

The symplectic Stiefel manifold is a Riemannian manifold that is a generalization of the symplectic group. In this study, we propose novel conjugate gradient methods on the symplectic Stiefel manifold and compare them with the steepest descent method proposed in existing studies through numerical experiments. Although the theoretical basis of the Riemannian conjugate gradient methods has already been established, special treatment is required to address specific manifolds since these methods utilize some mappings, such as a retraction and vector transport, on the manifold. Numerical experiments demonstrate that the proposed method outperforms existing methods and is efficient.

Gradient Descent Ascent for Minimax Problems on Riemannian Manifolds.

In the paper, we study a class of useful minimax problems on Riemanian manifolds and propose a class of effective Riemanian gradient-based methods to solve these minimax problems. Specifically, we propose an effective Riemannian gradient descent ascent (RGDA) algorithm for the deterministic minimax optimization. Moreover, we prove that our RGDA has a sample complexity of O(κ2ϵ-2) for finding an ϵ-stationary solution of the Geodesically-Nonconvex Strongly-Concave (GNSC) minimax problems, where κ denotes the condition number. At the same time, we present an effective Riemannian stochastic gradient descent ascent (RSGDA) algorithm for the stochastic minimax optimization, which has a sample complexity of O(κ4ϵ-4) for finding an ϵ-stationary solution. To further reduce the sample complexity, we propose an accelerated Riemannian stochastic gradient descent ascent (Acc-RSGDA) algorithm based on the momentum-based variance-reduced technique. We prove that our Acc-RSGDA algorithm achieves a lower sample complexity of ~O(κ4ϵ-3) in searching for an ϵ-stationary solution of the GNSC minimax problems. Extensive experimental results on the robust distributional optimization and robust Deep Neural Networks (DNNs) training over Stiefel manifold demonstrate efficiency of our algorithms.

Bisections of Mass Assignments Using Flags of Affine Spaces

We use recent extensions of the Borsuk–Ulam theorem for Stiefel manifolds to generalize the ham sandwich theorem to mass assignments. A k-dimensional mass assignment continuously imposes a measure on each k-dimensional affine subspace of $${\mathbb {R}}^d$$ . Given a finite collection of mass assignments of different dimensions, one may ask if there is some sequence of affine subspaces $$S_{k-1} \subset S_k \subset \ldots \subset S_{d-1} \subset {\mathbb {R}}^d$$ such that $$S_i$$ bisects all the mass assignments on $$S_{i+1}$$ for every i. We show it is possible to do so whenever the number of mass assignments of dimensions $$(k,\ldots ,d)$$ is a permutation of $$(k,\ldots ,d)$$ . We extend previous work on mass assignments and the central transversal theorem. We also study the problem of halving several families of $$(d-k)$$ -dimensional affine spaces of $${\mathbb {R}}^d$$ using a $$(k-1)$$ -dimensional affine subspace contained in some translate of a fixed k-dimensional affine space. For $$k=d-1$$ , there results can be interpreted as dynamic ham sandwich theorems for families of moving points.

Riemannian optimization with a preconditioning scheme on the generalized Stiefel manifold

Optimization problems on the generalized Stiefel manifold (and products of it) are prevalent across science and engineering. For example, in computational science they arise in symmetric (generalized) eigenvalue problems, in nonlinear eigenvalue problems, and in electronic structures computations, to name a few problems. In statistics and machine learning, they arise, for example, in various dimensionality reduction techniques such as canonical correlation analysis. In deep learning, regularization and improved stability can be obtained by constraining some layers to have parameter matrices that belong to the Stiefel manifold. Solving problems on the generalized Stiefel manifold can be approached via the tools of Riemannian optimization. However, using the standard geometric components for the generalized Stiefel manifold has two possible shortcomings: computing some of the geometric components can be too expensive and convergence can be rather slow in certain cases. Both shortcomings can be addressed using a technique called Riemannian preconditioning, which amounts to using geometric components derived by a preconditioner that defines a Riemannian metric on the constraint manifold. In this paper we develop the geometric components required to perform Riemannian optimization on the generalized Stiefel manifold equipped with a non-standard metric, and illustrate theoretically and numerically the use of those components and the effect of Riemannian preconditioning for solving optimization problems on the generalized Stiefel manifold.

Generalized left-localized Cayley parametrization for optimization with orthogonality constraints

We present a reformulation of optimization problems over the Stiefel manifold by using a Cayley-type transform, named the generalized left-localized Cayley transform, for the Stiefel manifold. The reformulated optimization problem is defined over a vector space, whereby we can apply directly powerful computational arts designed for optimization over a vector space. The proposed Cayley-type transform enjoys several key properties which are useful to (i) study relations between the original problem and the proposed problem; (ii) check the conditions to guarantee the global convergence of optimization algorithms. Numerical experiments demonstrate that the proposed algorithm outperforms the standard algorithms designed with a retraction on the Stiefel manifold.

ISOMETRY GROUPS OF GENERALIZED STIEFEL MANIFOLDS

A generalized Stiefel manifold is the manifold of orthonormal frames in a vector space with a nondegenerated bilinear or hermitian form. In this article, the Isometry group of the generalized Stiefel manifolds are computed at least up to connected components in an explicit form. This is done by considering a natural nonassociative algebra m associated to the affine structure of the Stiefel manifold, computing explicitly the automorphism group Aut(𝔪) and inducing this computation to the Isometry group.

A Riemannian stochastic representation for quantifying model uncertainties in molecular dynamics simulations

A Riemannian stochastic representation of model uncertainties in molecular dynamics is proposed. The approach relies on a reduced-order model, the projection basis of which is randomized on a subset of the Stiefel manifold characterized by a set of linear constraints defining, e.g., Dirichlet boundary conditions in the physical space. We first show that these constraints are, indeed, preserved through Riemannian pushforward and pullback actions to, and from, the tangent space to the manifold at any admissible point. This fundamental property is subsequently exploited to derive a probabilistic model that leverages the multimodel nature of the atomistic setting. The proposed formulation offers several advantages, including a simple and interpretable low-dimensional parameterization, the ability to constraint the Fréchet mean on the manifold, and ease of implementation and propagation. The relevance of the proposed modeling framework is finally demonstrated on various applications including multiscale simulations on graphene-based systems.

On generalizing trace minimization principles

Various trace minimization principles have interplayed with numerical computations for the standard eigenvalue and generalized eigenvalue problems in general, as well as important applied eigenvalue problems including the linear response eigenvalue problem from electronic structure calculation and the symplectic eigenvalue problem of positive definite matrices that play important roles in classical Hamiltonian dynamics, quantum mechanics, and quantum information, among others. In this paper, Ky Fan's trace minimization principle is extended along the line of the Brockett cost function tr(DXHAX) in X on the Stiefel manifold, where D of an apt size is positive definite. Specifically, we investigate infX⁡tr(DXHAX) subject to XHBX=Ik (the k×k identity matrix) or XHBX=Jk, where Jk=diag(±1). We establish conditions under which the infimum is finite and when it is finite, analytic solutions are obtained in terms of the eigenvalues and eigenvectors of the matrix pencil A−λB, where B is possibly indefinite and possibly singular, and D is also possibly indefinite.

Trace ratio optimization with an application to multi-view learning

A trace ratio optimization problem over the Stiefel manifold is investigated from the perspectives of both theory and numerical computations. Necessary conditions in the form of nonlinear eigenvalue problem with eigenvector dependency (NEPv) are established and a numerical method based on the self-consistent field (SCF) iteration with a postprocessing step is designed to solve the NEPv and the method is proved to be always convergent. As an application to multi-view subspace learning, a new framework and its instantiated concrete models are proposed and demonstrated on real world data sets. Numerical results show that the efficiency of the proposed numerical methods and effectiveness of the new orthogonal multi-view subspace learning models.

Disentangled Representation Learning and Generation With Manifold Optimization.

Disentanglement is a useful property in representation learning, which increases the interpretability of generative models such as variational autoencoders (VAE), generative adversarial models, and their many variants. Typically in such models, an increase in disentanglement performance is traded off with generation quality. In the context of latent space models, this work presents a representation learning framework that explicitly promotes disentanglement by encouraging orthogonal directions of variations. The proposed objective is the sum of an autoencoder error term along with a principal component analysis reconstruction error in the feature space. This has an interpretation of a restricted kernel machine with the eigenvector matrix valued on the Stiefel manifold. Our analysis shows that such a construction promotes disentanglement by matching the principal directions in the latent space with the directions of orthogonal variation in data space. In an alternating minimization scheme, we use the Cayley ADAM algorithm, a stochastic optimization method on the Stiefel manifold along with the Adam optimizer. Our theoretical discussion and various experiments show that the proposed model is an improvement over many VAE variants in terms of both generation quality and disentangled representation learning.

Omnidirectional precoding for massive MIMO with uniform rectangular array in presence of mutual coupling

The massive Multiple-Input and Multiple-Output (MIMO) system is one of the technologies in 6-Generation (6G) communication networks. Omnidirectional or broad coverage precoding is used in massive MIMO to broadcast the common signals from public channels to all active and non-active users so that the radiation power of these signals is constant in all spatial directions. On the other hand, in large-scale array antennas, the presence of mutual coupling leads to deteriorating the performance of MIMO systems. In this paper, we propose omnidirectional precoding of the massive MIMO equipped with a Uniform Rectangular Array (URA) in presence of the mutual coupling. In the proposed method, omnidirectional transmission and maximum achievable ergodic rate conditions are developed for the case that mutual coupling exists, then an omnidirectional precoding matrix is designed to satisfy three constraints: 1) Omnidirectional transmission, 2) Equal average power transmission per antenna, and 3) Maximum achievable ergodic rate. For this purpose, we show that an omnidirectional matrix can be reconstructed using a pair of vectors with elements' absolute value equal to one that forms a Complementary Set (CS), and the Complete Complementary Codes (CCC) that satisfied some conditions. The intended CS is selected from the Golay sequence and the intended CCC is obtained by solving a gradient algorithm with projection on the Grassmann and Stiefel manifolds. Simulation results confirm the expected results.

Multi-Manifold Optimization for Multi-View Subspace Clustering.

The meaningful patterns embedded in high-dimensional multi-view data sets typically tend to have a much more compact representation that often lies close to a low-dimensional manifold. Identification of hidden structures in such data mainly depends on the proper modeling of the geometry of low-dimensional manifolds. In this regard, this article presents a manifold optimization-based integrative clustering algorithm for multi-view data. To identify consensus clusters, the algorithm constructs a joint graph Laplacian that contains denoised cluster information of the individual views. It optimizes a joint clustering objective while reducing the disagreement between the cluster structures conveyed by the joint and individual views. The optimization is performed alternatively over k -means and Stiefel manifolds. The Stiefel manifold helps to model the nonlinearities and differential clusters within the individual views, whereas k -means manifold tries to elucidate the best-fit joint cluster structure of the data. A gradient-based movement is performed separately on the manifold of each view so that individual nonlinearity is preserved while looking for shared cluster information. The convergence of the proposed algorithm is established over the manifold and asymptotic convergence bound is obtained to quantify theoretically how fast the sequence of iterates generated by the algorithm converges to an optimal solution. The integrative clustering on benchmark and multi-omics cancer data sets demonstrates that the proposed algorithm outperforms state-of-the-art multi-view clustering approaches.

Energy-adaptive Riemannian optimization on the Stiefel manifold

This paper addresses the numerical solution of nonlinear eigenvector problems such as the Gross–Pitaevskii and Kohn–Sham equation arising in computational physics and chemistry. These problems characterize critical points of energy minimization problems on the infinite-dimensional Stiefel manifold. To efficiently compute minimizers, we propose a novel Riemannian gradient descent method induced by an energy-adaptive metric. Quantified convergence of the methods is established under suitable assumptions on the underlying problem. A non-monotone line search and the inexact evaluation of Riemannian gradients substantially improve the overall efficiency of the method. Numerical experiments illustrate the performance of the method and demonstrates its competitiveness with well-established schemes.

Fast and Efficient MMD-Based Fair PCA via Optimization over Stiefel Manifold

This paper defines fair principal component analysis (PCA) as minimizing the maximum mean discrepancy (MMD) between the dimensionality-reduced conditional distributions of different protected classes. The incorporation of MMD naturally leads to an exact and tractable mathematical formulation of fairness with good statistical properties. We formulate the problem of fair PCA subject to MMD constraints as a non-convex optimization over the Stiefel manifold and solve it using the Riemannian Exact Penalty Method with Smoothing (REPMS). Importantly, we provide a local optimality guarantee and explicitly show the theoretical effect of each hyperparameter in practical settings, extending previous results. Experimental comparisons based on synthetic and UCI datasets show that our approach outperforms prior work in explained variance, fairness, and runtime.

A Convex Relaxation Approach for Learning the Robust Koopman Operator

Although data-driven models, especially deep learning, have achieved astonishing results on many prediction tasks for nonlinear sequences, challenges still remain in finding an appropriate way to embed prior knowledge of physical dynamics in these models. In this work, we introduce a convex relaxation approach for learning robust Koopman operators of nonlinear dynamical systems, which are intended to construct approximate space spanned by eigenfunctions of the Koopman operator. Different from the classical dynamic mode decomposition, we use the layer weights of neural networks as eigenfunctions of the Koopman operator, providing intrinsic coordinates that globally linearize the dynamics. We find that the approximation of space can be regarded as an orthogonal Procrustes problem on the Stiefel manifold, which is highly sensitive to noise. The key contribution of this paper is to demonstrate that strict orthogonal constraint can be replaced by its convex relaxation, and the performance of the model can be improved without increasing the complexity when dealing with both clean and noisy data. After that, the overall model can be optimized via backpropagation in an end-to-end manner. The comparisons of the proposed method against several state-of-the-art competitors are shown on nonlinear oscillators and the lid-driven cavity flow.

Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs

The optimization with orthogonality has been shown useful in training deep neural networks (DNNs). To impose orthogonality on DNNs, both computational efficiency and stability are important. However, existing methods utilizing Riemannian optimization or hard constraints can only ensure stability while those using soft constraints can only improve efficiency. In this paper, we propose a novel method, named Feedback Gradient Descent (FGD), to our knowledge, the first work showing high efficiency and stability simultaneously. FGD induces orthogonality based on the simple yet indispensable Euler discretization of a continuous-time dynamical system on the tangent bundle of the Stiefel manifold. In particular, inspired by a numerical integration method on manifolds called Feedback Integrators, we propose to instantiate it on the tangent bundle of the Stiefel manifold for the first time. In our extensive image classification experiments, FGD comprehensively outperforms the existing state-of-the-art methods in terms of accuracy, efficiency, and stability.