Stiefel Manifold Research Articles

Manifold Neural Network With Non-Gradient Optimization.

Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent and thus has a slow convergence. In addition, softmax, as a decision layer, may ignore the distribution information of the data during classification. Aiming to tackle the referred problems, we propose a novel manifold neural network based on non-gradient optimization, i.e., the analytical-form solutions. Considering that the activation function is generally invertible, we reconstruct the network via forward ridge regression and low-rank backward approximation, which achieve rapid convergence. Moreover, by unifying the flexible Stiefel manifold and adaptive support vector machine, we devise the novel decision layer which efficiently fits the manifold structure of the data and label information. Consequently, a jointly non-gradient optimization method is designed to generate the network with analytical-form results. Furthermore, an acceleration strategy is utilize to reduce the time complexity for handling high dimensional datasets. Eventually, extensive experiments validate the superior performance of the model.

Emergent behaviors of high-dimensional Kuramoto models on Stiefel manifolds

In this paper, we study emergent dynamics for first and second-order high-dimensional Kuramoto type models on Stiefel manifolds. For the first-order consensus model on the Stiefel manifold proposed in Markdahl et al. (2018), we show that a homogeneous ensemble relaxes to completely consensus state exponentially fast. On the other hand for a heterogeneous ensemble, we provide sufficient conditions leading to the locked state in which relative distances between two states converge to definite values in a large coupling regime. We also propose a second-order extension of the first-order one, and study their emergent behaviors using Lyapunov functionals such as an energy functional and an averaged distance functional.

Nonlinear distributed state estimation on the Stiefel manifold using diffusion particle filters

Many relevant problems in engineering demand the estimation of dynamic system states that are constrained to non-Euclidean spaces. Furthermore, it is generally advantageous to implement estimation methods in a network of cooperating nodes instead of relying on a single data fusion center. With this in mind, we propose in this paper two new distributed particle filtering methods to track the states of a dynamic system that evolves on the Stiefel manifold, which arises naturally when the states are subject to nonlinear orthogonal constraints. The proposed algorithms are based on the Random Exchange and Adapt-then-Combine diffusion techniques, and perform parametric approximations using the matrix von Mises-Fisher distribution to compress information exchanged between network nodes. Estimates of the state are then determined via an empirical averaging method that approximates the centroid (Karcher mean) of the particle set. As we verify via numerical simulations, the proposed methods show improved performance compared to previous Particle and Extended Kalman filters designed for Euclidean state variables, and compared to a non-cooperative particle filtering algorithm.

An asymptotic thin shell condition and large deviations for random multidimensional projections

It is well known that fluctuations of marginals of high-dimensional random vectors that satisfy a certain concentration estimate called the thin shell condition are approximately Gaussian. In this article we identify a general condition on a sequence of high-dimensional random vectors under which one can identify the exponential decay rate of large deviation probabilities of the corresponding sequence of marginals. More precisely, consider the projection of an n-dimensional random vector onto a random kn-dimensional basis, kn≤n, drawn uniformly from the Haar measure on the Stiefel manifold of orthonormal kn-frames in Rn, in three different asymptotic regimes as n→∞: “constant” (kn=k), “sublinear” (kn→∞ but kn/n→0) and “linear” (kn/n→λ with 0<λ≤1). When the sequence of random vectors satisfies a certain “asymptotic thin shell condition”, we establish large deviation principles for the corresponding sequence of random projections in the constant regime, and for the sequence of empirical measures of the coordinates of the random projections in the sublinear and linear regimes. We also establish large deviation principles for scaled ℓq norms of the random projections in all three regimes. Moreover, we show that the asymptotic thin shell condition holds for various sequences of random vectors of interest, including the uniform measure on suitably scaled ℓpn balls, for p∈[1,∞), and generalized Orlicz balls defined via a superquadratic function, as well as a class of Gibbs measures with superquadratic interaction potential. Along the way, we obtain logarithmic asymptotics of volumes of high-dimensional Orlicz balls, which may be of independent interest. We also show that the decay rate of large deviation probabilities of Euclidean norms of multi-dimensional projections of ℓpn balls, when p∈[1,2), exhibits an unexpected phase transition in the sublinear regime, thus disproving an earlier conjecture due to Alonso-Gutiérrez et al. Random projections of high-dimensional random vectors are of interest in a range of fields including asymptotic convex geometry and high-dimensional statistics.

The Monogenic Hua–Radon Transform and Its Inverse

The monogenic Hua–Radon transform is defined as an orthogonal projection on holomorphic functions in the Lie sphere. Extending the work of Sabadini and Sommen (J Geom Anal 29:2709–2737, 2019), we determine its reproducing kernel. Integrating this kernel over the Stiefel manifold yields a linear combination of the zonal spherical monogenics. Using the reproducing properties of those monogenics, we obtain an inversion for the monogenic Hua–Radon transform.

Riemannian proximal stochastic gradient descent for sparse 2DPCA

Sparse two-dimensional principal component analysis (2DPCA) is an effective dimensionality reduction and data processing technique, which has been successfully applied in face recognition. Sparse 2DPCA is based on 2D image matrices instead of 1D vectors so it can well exploit the structural information of image pixels. The sparse 2DPCA model is regarded as a non-convex and non-smooth composite optimization problem. The manifold proximal gradient algorithm (ManPG) that is efficient and robust can globally converge to an optimal point in solving this kind of problem on the Stiefel manifold, such as sparse PCA. ManPG uses the proximal gradient method for the non-smooth term of this problem, which yields a convex subproblem on the tangent space. By solving this subproblem, a descent direction on the tangent space is obtained. Then a new update point on the manifold is obtained via retraction. However, in solving large-scale problems, the computational complexity of ManPG is quite high since ManPG requires calculating the full gradient in each iteration. In order to reduce the computational complexity of each iteration, a more computationally efficient Riemannian proximal stochastic gradient descent algorithm (RPSGD) is proposed, which utilizes the stochastic gradient instead of the full gradient in ManPG. We prove that the iteration complexity of the proposed RPSGD is O(1/ϵ4) while it maintains the sublinear convergence for the large-scale sparse 2DPCA. The experimental results on face recognition and handwritten digits classification data sets show RPSGD has the advantages of small-time overhead, high test accuracy, good reconstructed image quality, and relatively sparse solution in solving the large-scale sparse 2DPCA.

Improved time series clustering based on new geometric frameworks

Most existing methods for time series clustering rely on distances calculated from the entire raw data using the Euclidean distance or Dynamic Time Warping distance. In this work, we propose to embed the time series onto higher-dimensional spaces to obtain geometric representations of the time series themselves. Particularly, the embedding on Rn×p, on the Stiefel manifold and on the unit Sphere are analyzed for their performances with respect to several yet well-known clustering algorithms. The gain brought by the geometrical representation for the time series clustering is illustrated through a large benchmark of databases. We particularly exhibit that, firstly, the embedding of the time series on higher dimensional spaces gives better results than classical approaches and, secondly, that the embedding on the Stiefel manifold - in conjunction with UMAP and HDBSCAN clustering algorithms - is the recommended framework for time series clustering.

Second-order component analysis for fault detection

Process monitoring based on neural networks is getting more and more attention. Compared with classical neural networks, high-order neural networks have natural advantages in dealing with heteroscedastic data. However, high-order neural networks might bring the risk of overfitting, which learning both the key information from original data and noises or anomalies. Orthogonal constraints can greatly reduce correlations between extracted features, thereby reducing the overfitting risk. This paper proposes a novel fault detection method called second-order component analysis (SCA). SCA rules out the heteroscedasticity of process data by optimizing a second-order autoencoder with orthogonal constraints. In order to deal with this constrained optimization problem, a geometric conjugate gradient algorithm is adopted in this paper, which performs geometric optimization on the combination of Stiefel manifold and Euclidean manifold. Extensive experiments on the Tennessee-Eastman benchmark process show that SCA outperforms the compared state-of-the-art methods with missed detection rate (MDR) and false alarm rate (FAR).

A Manifold Proximal Linear Method for Sparse Spectral Clustering with Application to Single-Cell RNA Sequencing Data Analysis

Spectral clustering is one of the fundamental unsupervised learning methods and is widely used in data analysis. Sparse spectral clustering (SSC) imposes sparsity to the spectral clustering, and it improves the interpretability of the model. One widely adopted model for SSC in the literature is an optimization problem over the Stiefel manifold with nonsmooth and nonconvex objective. Such an optimization problem is very challenging to solve. Existing methods usually solve its convex relaxation or need to smooth its nonsmooth objective using certain smoothing techniques. Therefore, they were not targeting solving the original formulation of SSC. In this paper, we propose a manifold proximal linear method (ManPL) that solves the original SSC formulation without twisting the model. We also extend the algorithm to solve multiple-kernel SSC problems, for which an alternating ManPL algorithm is proposed. Convergence and iteration complexity results of the proposed methods are established. We demonstrate the advantage of our proposed methods over existing methods via clustering of several data sets, including University of California Irvine and single-cell RNA sequencing data sets.

An efficient damped Newton-type algorithm with globalization strategy on Riemannian manifolds

We propose a new globalization strategy of the damped Newton method for finding singularities of a vector field on Riemannian manifolds. We establish its global convergence with a superlinear rate. In particular, this globalization generalizes the known damped Newton’s method for general retraction. The global convergence analysis presented here does not require any hypothesis regarding a singularity of the vector field. We applied the proposed method to solve the truncated singular value problem on the product of two Stiefel manifolds, the dextrous hand grasping problem and an academic problem on the cone of symmetric positive definite matrices, the Rayleigh quotient and a problem of finding zeros of a non-conservative vector field on the sphere. Numerical experiments are presented, showing that the proposed algorithm is more robust than the known damped Newton’s method.

The λ-cosine transforms, differential operators, and Funk transforms on Stiefel and Grassmann manifolds

We introduce a new family of invariant differential operators associated with λ-cosine and Funk-Radon transforms on Stiefel and Grassmann manifolds. These operators reduce the order of the λ-cosine transforms and yield new inversion formulas. Intermediate Funk-cosine transforms corresponding to integration over matrices of lower rank are studied. The main tools are polar decomposition and Fourier analysis on matrix space.

A nonparametric probabilistic method to enhance PGD solutions with data-driven approach, application to the automated tape placement process

A nonparametric method assessing the error and variability margins in solutions depicted in a separated form using experimental results is illustrated in this work. The method assess the total variability of the solution including the modeling error and the truncation error when experimental results are available. The illustrated method is based on the use of the PGD separated form solutions, enriched by transforming a part of the PGD basis vectors into probabilistic one. The constructed probabilistic vectors are restricted to the physical solution’s Stiefel manifold. The result is a real-time parametric PGD solution enhanced with the solution variability and the confidence intervals.

Convolutional proximal neural networks and Plug-and-Play algorithms

In this paper, we introduce convolutional proximal neural networks (cPNNs), which are by construction averaged operators. For filters with full length, we propose a stochastic gradient descent algorithm on a submanifold of the Stiefel manifold to train cPNNs. In case of filters with limited length, we design algorithms for minimizing functionals that approximate the orthogonality constraints imposed on the operators by penalizing the least squares distance to the identity operator. Then, we investigate how scaled cPNNs with a prescribed Lipschitz constant can be used for denoising signals and images, where the achieved quality depends on the Lipschitz constant. Finally, we apply cPNN based denoisers within a Plug-and-Play framework and provide convergence results for the corresponding PnP forward-backward splitting algorithm based on an oracle construction.

HSH-norm optimal MOR for the MIMO linear time-invariant systems on the Stiefel manifold

This paper focuses on the Hilbert–Schmidt-Hankel-norm ( norm) optimal model order reduction (MOR) of large-scale multiple-input multiple-output (MIMO) linear time-invariant (LTI) systems on the Stiefel manifold. First, a cost function is constructed in regard to the norm. By introducing the orthogonality constraints, the norm optimal MOR problem is converted into an unconstrained minimisation problem on the Stiefel manifold. We derive the Riemannian gradient of the cost function on the Stiefel manifold. The MOR algorithm is developed associated with the Riemannian conjugate gradient method. Global convergence is guaranteed with mild conditions. Finally, the effectiveness of the proposed method is illustrated by two numerical examples.

Inversions for the Hua-Radon and the polarized Hua-Radon transform

The Hua-Radon and polarized Hua-Radon transform are two orthogonal projections defined on holomorphic functions in the Lie sphere. Both transformations can be written as integral transforms with respect to a suitable reproducing kernel. Integrating both kernels over a Stiefel manifold yields a linear combination of zonal spherical monogenics. Using an Almansi type decomposition of holomorphic functions and reproducing properties of the zonal monogenics, we obtain an inversion formula for both the Hua-Radon and the polarized Hua-Radon transform.

Geodesic orbit metrics in a class of homogeneous bundles over real and complex Stiefel manifolds

Geodesic orbit spaces (or g.o. spaces) are defined as those homogeneous Riemannian spaces \((M=G/H,g)\) whose geodesics are orbits of one-parameter subgroups of G. The corresponding metric g is called a geodesic orbit metric. We study the geodesic orbit spaces of the form (G/H, g), such that G is one of the compact classical Lie groups \({{\,\mathrm{SO}\,}}(n)\), \(\mathrm{U}(n)\), and H is a diagonally embedded product \(H_1\times \cdots \times H_s\), where \(H_j\) is of the same type as G. This class includes spheres, Stiefel manifolds, Grassmann manifolds and real flag manifolds. The present work is a contribution to the study of g.o. spaces (G/H, g) with H semisimple.

A non-intrusive space-time interpolation from compact Stiefel manifolds of parametrized rigid-viscoplastic FEM problems

This work aims to interpolate parametrized reduced order model (ROM) basis constructed via the proper orthogonal decomposition (POD) to derive a robust ROM of the system’s dynamics for an unseen target parameter value. A novel non-intrusive space-time (ST) POD basis interpolation scheme is proposed, for which we define ROM spatial and temporal basis curves on compact Stiefel manifolds. An interpolation is finally defined on a mixed part encoded in a square matrix directly deduced using the spacial part, the singular values and the temporal part, to obtain an interpolated snapshot matrix, keeping track of accurate space and temporal eigenvectors. Moreover, in order to establish a well-defined curve on the compact Stiefel manifold, we introduce a new procedure, the so-called oriented SVD. Such an oriented SVD produces unique right and left eigenvectors for generic matrices, for which all singular values are distinct. It is important to notice that the ST POD basis interpolation does not require the construction and the subsequent solution of a reduced-order FEM model as classically is done. Hence it is avoiding the bottleneck of standard POD interpolation which is associated with the evaluation of the nonlinear terms of the Galerkin projection on the governing equations. As a proof of concept, the proposed method is demonstrated with the adaptation of rigid-thermoviscoplastic finite element ROMs applied to a typical nonlinear open forging metal forming process. Strong correlations of the ST POD models with respect to their associated high-fidelity FEM counterpart simulations are reported, highlighting its potential use for near real-time parametric simulations using off-line computed ROM POD databases.

Accelerated Optimization with Orthogonality Constraints

We develop a generalization of Nesterov's accelerated gradient descent method which is designed to deal with orthogonality constraints. To demonstrate the effectiveness of our method, we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number, and also compare with existing state-of-the-art quasi-Newton methods on the Stiefel manifold. Our experiments show that our method outperforms existing state-of-the-art quasi-Newton methods on some large, ill-conditioned problems.

Clustering of cancer data based on Stiefel manifold for multiple views

BackgroundIn recent years, various sequencing techniques have been used to collect biomedical omics datasets. It is usually possible to obtain multiple types of omics data from a single patient sample. Clustering of omics data plays an indispensable role in biological and medical research, and it is helpful to reveal data structures from multiple collections. Nevertheless, clustering of omics data consists of many challenges. The primary challenges in omics data analysis come from high dimension of data and small size of sample. Therefore, it is difficult to find a suitable integration method for structural analysis of multiple datasets.ResultsIn this paper, a multi-view clustering based on Stiefel manifold method (MCSM) is proposed. The MCSM method comprises three core steps. Firstly, we established a binary optimization model for the simultaneous clustering problem. Secondly, we solved the optimization problem by linear search algorithm based on Stiefel manifold. Finally, we integrated the clustering results obtained from three omics by using k-nearest neighbor method. We applied this approach to four cancer datasets on TCGA. The result shows that our method is superior to several state-of-art methods, which depends on the hypothesis that the underlying omics cluster class is the same.ConclusionParticularly, our approach has better performance than compared approaches when the underlying clusters are inconsistent. For patients with different subtypes, both consistent and differential clusters can be identified at the same time.

Equigeodesics on some classes of homogeneous spaces

We study homogeneous curves on some classes of reductive homogeneous spaces G/H which are geodesics with respect to any G-invariant metric on G/H. These curves are called equigeodesics. The spaces we consider are certain Stiefel manifolds VkRn, generalized Wallach spaces and spheres. We give a characterization for algebraic equigeodesics on V2Rn, V4R6, SO(6)/SO(3)⋅SO(2), W6=U(3)/U(1)3, W12=Sp(3)/Sp(1)3, S2n+1≅U(n+1)/U(n) and S4n+3≅Sp(n+1)/Sp(n).