Stiefel Manifold Research Articles

Riemannian gradient methods for stochastic composition problems

In the paper, we study a class of novel stochastic composition optimization problems over Riemannian manifold, which have been raised by multiple emerging machine learning applications such as distributionally robust learning in Riemannian manifold setting. To solve these composition problems, we propose an effective Riemannian compositional gradient (RCG) algorithm, which has a sample complexity of O(ϵ−4) for finding an ϵ-stationary point. To further reduce sample complexity, we propose an accelerated momentum-based Riemannian compositional gradient (M-RCG) algorithm. Moreover, we prove that the M-RCG obtains a lower sample complexity of Õ(ϵ−3) without large batches, which achieves the best known sample complexity for its Euclidean counterparts. Extensive numerical experiments on training deep neural networks (DNNs) over Stiefel manifold and learning principal component analysis (PCA) over Grassmann manifold demonstrate effectiveness of our proposed algorithms. To the best of our knowledge, this is the first study of the composition optimization problems over Riemannian manifold.

Computing the Riemannian Logarithm on the Stiefel Manifold: Metrics, Methods, and Performance

We address the problem of computing Riemannian normal coordinates on the real, compact Stiefel manifold of orthonormal frames. The Riemannian normal coordinates are based on the so-called Riemannian exponential and the associated Riemannian logarithm map and enable one to transfer almost any computational procedure to the realm of the Stiefel manifold. To compute the Riemannian logarithm is to solve the (local) geodesic endpoint problem. Instead of restricting the consideration to geodesics with respect to a single selected metric, we consider a family of Riemannian metrics introduced by Hüper, Markina, and Silva Leite that includes the Euclidean and the canonical metric as prominent examples. As main contributions, we provide (1) a unified, structured, reduced formula for the Stiefel geodesics. The formula is unified in the sense that it works for the full family of metrics under consideration. It is structured in the sense that it relies on matrix exponentials of skew-symmetric matrices exclusively. It is reduced in relation to the dimension of the matrices of which matrix exponentials have to be calculated. We provide (2) a unified method to tackle the geodesic endpoint problem numerically, and (3) we improve the existing Riemannian log algorithm under the canonical metric in terms of the computational efficiency. The findings are illustrated by means of numerical examples, where the novel algorithms prove to be the most efficient methods known to this date.

Estimates of covering type and minimal triangulations based on category weight

Abstract In a recent publication [D. Govc, W. A. Marzantowicz and P. Pavešić, Estimates of covering type and the number of vertices of minimal triangulations, Discrete Comput. Geom. 63 2020, 1, 31–48], we have introduced a new method, based on the Lusternik–Schnirelmann category and the cohomology ring of a space X, that yields lower bounds for the size of a triangulation of X. In this current paper, we present an important extension that takes into account the fundamental group of X. In fact, if π 1 ⁢ ( X ) {\pi_{1}(X)} contains elements of finite order, then one can often find cohomology classes of high ‘category weight’, which in turn allow for much stronger estimates of the size of triangulations of X. We develop several weighted estimates and then apply our method to compute explicit lower bounds for the size of triangulations of orbit spaces of cyclic group actions on a variety of spaces including products of spheres, Stiefel manifolds, Lie groups and highly-connected manifolds.

Image feature selection based on orthogonal [formula omitted] norms

This study presents a feature selection method based on orthogonal ℓ2,0-norms to reduce dimensions, especially for images, where correlated and redundant information is frequently present by nature. Recent ℓ2,0-norm methods have shown a way of discovering sparsity, but redundant features could still be selected in the process. In light of such, this study considers imposing an orthogonal constraint on sparsity, further limiting ℓ2,0 norms. To such an end, projection onto Stiefel manifolds is computed to satisfy the orthogonal constraint while ℓ2,0-norm regularization is computed via ℓp-box zero-one programming. Experiments on open datasets were carried out to evaluate the proposed method and different models, including ℓ1-, ℓ2,1-, and ℓ2,0-norm approaches. The experimental results showed that the mean accuracy and F1 scores of the proposed method were higher than those of the ℓ2,0-norm method without orthogonal constraints and those of the other baselines, subsequently proving the effectiveness of the proposed idea.

On the generalized trace ratio problem

The generalized trace ratio problem (GTRP) is to maximize the quotient over the Stiefel manifold. We establish the first-order necessary and sufficient optimality conditions for the global maximizer. Based on the classical second-order necessary optimality condition, we prove that GTRP has no local non-global maximizer. Finally we reveal the hidden convexity property of GTRP by applying the convex hull representation of the orthogonal similarity set.

Accelerated Optimization on Riemannian Manifolds via Discrete Constrained Variational Integrators

A variational formulation for accelerated optimization on normed vector spaces was recently introduced in Wibisono et al. (PNAS 113:E7351–E7358, 2016), and later generalized to the Riemannian manifold setting in Duruisseaux and Leok (SJMDS, 2022a). This variational framework was exploited on normed vector spaces in Duruisseaux et al. (SJSC 43:A2949–A2980, 2021) using time-adaptive geometric integrators to design efficient explicit algorithms for symplectic accelerated optimization, and it was observed that geometric discretizations which respect the time-rescaling invariance and symplecticity of the Lagrangian and Hamiltonian flows were substantially less prone to stability issues, and were therefore more robust, reliable, and computationally efficient. As such, it is natural to develop time-adaptive Hamiltonian variational integrators for accelerated optimization on Riemannian manifolds. In this paper, we consider the case of Riemannian manifolds embedded in a Euclidean space that can be characterized as the level set of a submersion. We will explore how holonomic constraints can be incorporated in discrete variational integrators to constrain the numerical discretization of the Riemannian Hamiltonian system to the Riemannian manifold, and we will test the performance of the resulting algorithms by solving eigenvalue and Procrustes problems formulated as optimization problems on the unit sphere and Stiefel manifold.

Riemannian optimization approach to structure-preserving model order reduction of integral-differential systems on the product of two Stiefel manifolds

In this paper, we discuss the structure-preserving H2 optimal model order reduction (MOR) problem of integral-differential systems based on the projection technique. Different from the common idea, we take a new way to deal with this minimization problem and for the first time reduce the integral-differential systems on the product of two Stiefel manifolds. The cost function is viewed as a function whose parameters are a pair of projection matrices with different dimensions, and then the H2 optimal MOR problem is reformulated as the minimization problem of the product of two Stiefel manifolds. We extend the Riemannian geometry of the single Stiefel manifold to the product manifold and the Riemannian gradient of the cost function is derived using the orthogonal projection of the product manifold. Further, we design a Riemannian conjugate gradient scheme on the product manifold, and propose a structure-preserving Riemannian conjugate gradient algorithm. The resulting search direction is a descent direction of the cost function. Besides, we show that our algorithm is globally convergent and has the potential application to second-order systems. Finally, numerical examples demonstrate the effectiveness of the proposed algorithm.

Coordinate Descent Without Coordinates: Tangent Subspace Descent on Riemannian Manifolds

We extend coordinate descent to manifold domains and provide convergence analyses for geodesically convex and nonconvex smooth objective functions. Our key insight is to draw an analogy between coordinate blocks in Euclidean space and tangent subspaces of a manifold. Hence, our method is called tangent subspace descent (TSD). The core principle behind ensuring convergence of TSD is the appropriate choice of subspace at each iteration. To this end, we propose two novel conditions, the (C, r)-norm and C-randomized norm conditions on deterministic and randomized modes of subspace selection, respectively, that promise convergence for smooth functions and that are satisfied in practical contexts. We propose two subspace selection rules, one deterministic and another randomized, of particular practical interest on the Stiefel manifold. Our proof-of-concept numerical experiments on the sparse principal component analysis problem demonstrate TSD’s efficacy. Funding: This work was supported by the National Science Foundation [Grant 1740707] and the Defense Advanced Research Projects Agency Lagrange Program [Grant N660011824020].

BP-cohomology of projective Stiefel manifolds

In this paper, we compute the $BP$-cohomology of complex projective Stiefel manifolds. The method involves the homotopy fixed point spectral sequence, and works for complex oriented cohomology theories. We also use these calculations and $BP$-operations to prove new results about equivariant maps between Stiefel manifolds.

A dual pair for the contact group

AbstractGeneralizing the canonical symplectization of contact manifolds, we construct an infinite dimensional non-linear Stiefel manifold of weighted embeddings into a contact manifold. This space carries a symplectic structure such that the contact group and the group of reparametrizations act in a Hamiltonian fashion with equivariant moment maps, respectively, giving rise to a dual pair, called the EPContact dual pair. Via symplectic reduction, this dual pair provides a conceptual identification of non-linear Grassmannians of weighted submanifolds with certain coadjoint orbits of the contact group. Moreover, the EPContact dual pair gives rise to singular solutions for the geodesic equation on the group of contact diffeomorphisms. For the projectivized cotangent bundle, the EPContact dual pair is closely related to the EPDiff dual pair due to Holm and Marsden.

On the cohomology rings of partially projective quaternionic Stiefel manifolds

Abstract The quaternionic Stiefel manifold is the total space of a fibre bundle over the corresponding Grassmannian . The group acts freely on the fibres of this bundle. The quotient space is called the quaternionic projective Stiefel manifold. Its real and complex analogues were actively studied earlier by a number of authors. A finite group acting freely on the three-dimensional sphere also acts freely and discretely on the fibres of the quaternionic Stiefel bundle. The corresponding quotient spaces are called partially projective Stiefel manifolds. The cohomology rings of partially projective quaternionic Stiefel manifolds with coefficients in , where is prime, are calculated. Bibliography: 14 titles.

Nested Grassmannians for Dimensionality Reduction with Applications

In the recent past, nested structure of Riemannian manifolds has been studied in the context of dimensionality reduction as an alternative to the popular principal geodesic analysis (PGA) technique, for example, the principal nested spheres. In this paper, we propose a novel framework for constructing a nested sequence of homogeneous Riemannian manifolds. Common examples of homogeneous Riemannian manifolds include the spheres, the Stiefel manifolds, and the Grassmann manifolds. In particular, we focus on applying the proposed framework to the Grassmann manifolds, giving rise to the nested Grassmannians (NG). An important application in which Grassmann manifolds are encountered is planar shape analysis. Specifically, each planar (2D) shape can be represented as a point in the complex projective space which is a complex Grassmann manifold. Some salient features of our framework are: (i) it explicitly exploits the geometry of the homogeneous Riemannain manifolds and (ii) the nested lower-dimensional submanifolds need not be geodesic. With the proposed NG structure, we develop algorithms for the supervised and unsupervised dimensionality reduction problems respectively. The proposed algorithms are compared with PGA via simulation studies and real data experiments and are shown to achieve a higher ratio of expressed variance compared to PGA.<br>The code is available at <a href='https://github.com/cvgmi/NestedGrassmann'>https://github.com/cvgmi/NestedGrassmann</a>

Stochastic consensus dynamics for nonconvex optimization on the Stiefel manifold: Mean-field limit and convergence

We study a consensus-based method for minimizing a nonconvex function over the Stiefel manifold. The consensus dynamics consists of stochastic differential equations for interacting particle system, whose trajectory is guaranteed to stay on the Stiefel manifold. For the proposed model, we prove the mean-field limit of the stochastic system toward a nonlinear Fokker–Planck equation on the Stiefel manifold. Moreover, we provide a sufficient condition on the parameter and the initial data, so that the solution to the Fokker–Planck equation is asymptotically concentrated on the point near a global optimizer. To implement our consensus-based optimization (CBO) algorithm, we provide two algorithms; one is improved from the algorithm suggested in our previous work, and the other is based on an entirely different approach, namely the Cayley transformation. We validate the CBO algorithms on the various test problems on the Stiefel manifold.

Closed-form Geodesics and Optimization for Riemannian Logarithms of Stiefel and Flag Manifolds

Closed-form Geodesics and Optimization for Riemannian Logarithms of Stiefel and Flag Manifolds

Optimization flows landing on the Stiefel manifold⋆

We study a continuous-time system that solves optimization problems over the set of orthonormal matrices, which is also known as the Stiefel manifold. The resulting optimization flow follows a path that is not always on the manifold but asymptotically lands on the manifold. We introduce a generalized Stiefel manifold to which we extend the canonical metric of the Stiefel manifold. We show that the vector field of the proposed flow can be interpreted as the sum of a Riemannian gradient on a generalized Stiefel manifold and a normal vector. Moreover, we prove that the proposed flow globally converges to the set of critical points, and any local minimum and isolated critical point is asymptotically stable.

Decentralized Optimization Over the Stiefel Manifold by an Approximate Augmented Lagrangian Function

In this paper, we focus on the decentralized optimization problem over the Stiefel manifold, which is defined on a connected network of <inline-formula><tex-math notation="LaTeX">$d$</tex-math></inline-formula> agents. The objective is an average of <inline-formula><tex-math notation="LaTeX">$d$</tex-math></inline-formula> local functions, and each function is privately held by an agent and encodes its data. The agents can only communicate with their neighbors in a collaborative effort to solve this problem. In existing methods, multiple rounds of communications are required to guarantee the convergence, giving rise to high communication costs. In contrast, this paper proposes a decentralized algorithm, called DESTINY, which only invokes a single round of communications per iteration. DESTINY combines gradient tracking techniques with a novel approximate augmented Lagrangian function. The global convergence to stationary points is rigorously established. Comprehensive numerical experiments demonstrate that DESTINY has a strong potential to deliver a cutting-edge performance in solving a variety of testing problems.

Nonlinear Orthogonal NMF on the Stiefel Manifold With Graph-Based Total Variation Regularization

This paper proposes a novel Nonlinear Orthogonal NMF model with Graph-based Total Variation regularization (GTV) for Multispectral document images decomposition. In this model, a GTV regularization is incorporated to preserve the intrinsic geometrical structure of document content lost by the vectorization of spectral images. A spatial orthogonality constraint over the Stiefel manifold is imposed to improve the sparsity of the solution and ensure its uniqueness. The kernel trick is involved to account for the non-linear correlation inherent to spectral data. We devised an efficient algorithm to solve the formulated problem using the Alternating Direction Method of Multipliers (ADMM). The experimental results on real-world data show that the proposed model achieves better decomposition performance than recent competitive methods and outperforms some traditional state-of-the-art methods.

Slow and Finite-Time Relaxations to M-Bipartite Consensus on the Stiefel Manifold

Slow and Finite-Time Relaxations to M-Bipartite Consensus on the Stiefel Manifold

Variational Bayesian Orthogonal Nonnegative Matrix Factorization Over the Stiefel Manifold.

Nonnegative matrix factorization (NMF) is one of the best-known multivariate data analysis techniques. The NMF uniqueness and its rank selection are two major open problems in this field. The solutions uniqueness issue can be addressed by imposing the orthogonality condition on NMF. This constraint yields sparser part-based representations and improved performance in clustering and source separation tasks. However, existing orthogonal NMF algorithms rely mainly on non-probabilistic frameworks that ignore the noise inherent in real-life data and lack variable uncertainties. Thus, in this work, we investigate a new probabilistic formulation of orthogonal NMF (ONMF). In the proposed model, we impose the orthogonality through a directional prior distribution defined on the Stiefel manifold called von Mises-Fisher distribution. This manifold consists of a set of directions that comply with the orthogonality condition that arises in many applications. Moreover, our model involves an automatic relevance determination (ARD) prior to address the model order selection issue. We devised an efficient variational Bayesian inference algorithm to solve the proposed ONMF model, which allows fast processing of large datasets. We evaluated the proposed model, called VBONMF, on the task of blind decomposition of real-world multispectral images of ancient documents. The numerical experiments demonstrate its efficiency and competitiveness compared to the state-of-the-art approaches.

Probabilistic Principal Geodesic Deep Metric Learning

Similarity learning which is useful for the purpose of comparing various characteristics of images in the computer vision field has been often applied for deep metric learning (DML). Also, a lot of combinations of pairwise similarity metrics such as Euclidean distance and cosine similarity have been studied actively. However, such a local similarity-based approach can be rather a bottleneck for a retrieval task in which global characteristics of images must be considered important. Therefore, this paper proposes a new similarity metric structure that considers the local similarity as well as the global characteristic on the representation space, i.e., class variability. Also, based on an insight that better class variability analysis can be accomplished on the Stiefel (or Riemannian) manifold, manifold geometry is employed to generate class variability information. Finally, we show that the proposed method designed through in-depth analysis of generalization bound of DML outperforms conventional DML methods theoretically and experimentally.