Singular Subspaces Research Articles

On perturbations for spectrum and singular value decompositions followed by deflation techniques

The calculation of the dominant eigenvalues of a symmetric matrix A together with its eigenvectors, followed by the calculation of the deflation of A1=A−ρUkUkT corresponds to one step of the Wielandt deflation technique, where ρ is a shift and Uk are eigenvectors of A. In this paper, we investigate how the eigenspace of A1 changes when A1 is perturbed to A˜1=A−ρU˜kU˜kT, where U˜k are approximate eigenvectors of A. We establish the bounds for the angle of eigenspaces of A1 and A˜1 based on the Davis-Kahan theorem. Moreover, in the practical implementation for singular value decomposition, once one or several singular triplets converge to a preset accuracy, they should be deflated by B1=B−γWkVkH with γ being a shift, Wk and Vk are singular vectors of B, so that they will not be re-computed. We investigate how the singular subspaces of B1=B−γWkVkH change when B1 is perturbed to B˜1=B−γW˜kV˜kH, W˜k and V˜k are approximate singular vectors of B. We also establish the bounds for the angle of singular subspaces of B1 and B˜1 based on the Wedin theorem.

Probabilistic Perturbation Bounds for Invariant, Deflating and Singular Subspaces

In this paper, we derive new probabilistic bounds on the sensitivity of invariant subspaces, deflation subspaces and singular subspaces of matrices. The analysis exploits a unified method for deriving asymptotic perturbation bounds of the subspaces under interest and utilizes probabilistic approximations of the entries of random perturbation matrices implementing the Markoff inequality. As a result of the analysis, we determine with a prescribed probability asymptotic perturbation bounds on the angles between the corresponding perturbed and unperturbed subspaces. It is shown that the probabilistic asymptotic bounds proposed are significantly less conservative than the corresponding deterministic perturbation bounds. The results obtained are illustrated by examples comparing the known deterministic perturbation bounds with the new probabilistic bounds.

Componentwise Perturbation Analysis of the Singular Value Decomposition of a Matrix

A rigorous perturbation analysis is presented for the singular value decomposition (SVD) of a real matrix with full column rank. It is proved that the SVD perturbation problem is well posed only when the singular values are distinct. The analysis involves the solution of symmetric coupled systems of linear equations. It produces asymptotic (local) componentwise perturbation bounds on the entries of the orthogonal matrices participating in the decomposition of the given matrix and on its singular values. Local bounds are derived for the sensitivity of the singular subspaces measured by the angles between the unperturbed and perturbed subspaces. Determining the asymptotic bounds of the orthogonal matrices and the sensitivity of singular subspaces requires knowing only the norm of the perturbation of the given matrix. An iterative scheme is described to find global bounds on the respective perturbations, and results from numerical experiments are presented.

The condition number of singular subspaces, revisited

I revisit the condition number of computing left and right singular subspaces from J.-G. Sun (1996) [19]. For real and complex matrices, I present an alternative computation of this condition number in the Euclidean distance on the input space of matrices and the chordal, Grassmann, and Procrustes distances on the output Grassmannian manifold of linear subspaces. Up to a small factor, this condition number equals the inverse minimum singular value gap between the singular values corresponding to the selected singular subspace and those not selected.

Covariance Analysis of the Estimated Markov Parameters in a Subspace Identification Method

It is important to provide a covariance of the estimates to ensure the quality of the identification results. This paper proposes a covariance of the estimated Markov parameters in a method previously proposed by the authors. The proposed covariance uses the gap between singular subspaces to estimate the perturbation of the extended observability matrix. The gap based analysis gives a simple expression in the sense that the estimated error in the singular subspace is strictly linear with respect to the perturbation of the original matrix and the left and right singular vectors. Numerical simulation shows the validity of the proposed covariance of the estimated Markov parameters.

A lightweight stochastic subspace identification-based modal parameters identification method of time-varying structural systems

The application of stochastic subspace identification (SSI) on identifying structural time-varying modal parameters suffers from the shortcomings of poor computational efficiency and susceptibility to spurious modes. This work overcomes the shortcomings by the following procedures based on existing covariance-driven SSI. First, the covariance-driven SSI is embedded with techniques of randomized singular value decomposition and subspace iteration for reducing computation cost and avoiding accuracy loss, forming the lightweight SSI (lwSSI). Then, the lwSSI is combined with the technique of sliding window for eliminating spurious modes and continuously identifying time-varying modal parameters. The proposed lwSSI-based identification method is applied on the simulation of a gravity dam, and the identified time-varying modal parameters show the same trend as the assumed time-varying elastic modules. Furthermore, the characteristics of the lwSSI-based identification method, including noise resistance, computational efficiency, and identification accuracy, are investigated. Finally, the identification of a steel cantilever beam with the time-varying modal parameters caused by the mass distribution experimentally validates the effectiveness of the proposed lwSSI-based identification method.

Classical and Quantized Maxwell Fields Deduced from Algebraic Many-Photon Theory

The deduction starts with the (non-relativistic) one-photon Hilbert space \mathcal{H} , equipped with the one-photon Hamiltonian and basic symmetry generators, as the only input information. We recall in the functorially associated boson Fock space the multi-photon dynamics and symmetry transformations, as well as the field operator (as the scaled self-adjoint part of the creation operator) and the q(uasi)-classical states. There is no reference to a presupposed classical Maxwell theory. By abstraction, we go over to the algebraic formulation of the multi-photon theory in terms of a C*-Weyl algebra. Its test function space E\subset \mathcal{H} is constructed as a nuclear Fréchet space, in which – via infrared damping – the dynamics and symmetries are nuclear continuous and their generators bounded. Each w*-closed, singular subspace of the continuous dual E' determines non-Fock coherent states and their mixtures lead to a representation von Neumann algebra with non-trivial center. The symmetry generators restricted to the center can be transformed into the Maxwell form by means of a symplectic transformation and involve the well-known conservation quantities of electrodynamics. This identifies the central part of the represented photon field operator as composed of the two classical canonical electrodynamic field components. We have obtained, therefore, in free space a kind of fusion of the multi-photon theory and the Maxwell theory of transverse electrodynamic fields, where the latter arise as derived quantities. By means of a Bogoliubov transformation one also gets a fusion of the quantized with the classical Maxwell theory, deduced from the photon concept. A sketch of non-relativistic gauging is added in the appendix to gain longitudinal, cohomological, and scalar potentials.

An essay on Freudenthal-Tits polar spaces

We study some properties of the nonembeddable polar spaces related to octonion division rings (and related to the index E7,328). More exactly, we classify its subspaces and show that its generating rank is equal to 5, whereas it was always believed to be at least 6. We also study self-projectivities of length 3 in the maximal singular subspaces, and these turn out to be polarities of octonion projective planes. We establish a connection between the conjugacy classes of such polarities and the orbits of the collineation group of the polar space on triples of opposite singular planes. Along the way, we classify all polar spaces for which each self-projectivity of length 3 in a maximal singular subspace is a polarity.

On Soft Normed Quasilinear Spaces

In this study, we investigate some properties of soft quasi-sequences and present new results. We then study the completeness of soft normed quasilinear space and present an analog of convergence and boundness results of soft quasi sequences in soft normed quasilinear spaces. Moreover, we define regular and singular subspaces of soft quasilinear spaces and draw several conclusions related to these notions. Afterward, we provide examples of these results in soft normed quasilinear spaces generalizing well-known results in soft linear normed spaces. Additionally, we offer new results concerning soft quasi subspaces of soft normed quasilinear spaces. Finally, we discuss the need for further research.

Continuous chiral distances for two-dimensional lattices.

Chirality was traditionally considered a binary property of periodic lattices and crystals. However, the classes of two-dimensional lattices modulo rigid motion form a continuous space, which was recently parametrized by three geographic-style coordinates. The four non-oblique Bravais classes of two-dimensional lattices form low-dimensional singular subspaces in the full continuous space. Now, the deviations of a lattice from its higher symmetry neighbors can be continuously quantified by real-valued distances satisfying metric axioms. This article analyzes these and newer G-chiral distances for millions of two-dimensional lattices that are extracted from thousands of available two-dimensional materials and real crystal structures in the Cambridge Structural Database.

Unsupervised anomaly detection via two-dimensional singular value decomposition and subspace reconstruction for multivariate time series

Unsupervised anomaly detection via two-dimensional singular value decomposition and subspace reconstruction for multivariate time series

Inference for low-rank tensors—no need to debias

In this paper, we consider the statistical inference for several low-rank tensor models. Specifically, in the Tucker low-rank tensor PCA or regression model, provided with any estimates achieving some attainable error rate, we develop the data-driven confidence regions for the singular subspace of the parameter tensor based on the asymptotic distribution of an updated estimate by two-iteration alternating minimization. The asymptotic distributions are established under some essential conditions on the signal-to-noise ratio (in PCA model) or sample size (in regression model). If the parameter tensor is further orthogonally decomposable, we develop the methods and nonasymptotic theory for inference on each individual singular vector. For the rank-one tensor PCA model, we establish the asymptotic distribution for general linear forms of principal components and confidence interval for each entry of the parameter tensor. Finally, numerical simulations are presented to corroborate our theoretical discoveries. In all of these models, we observe that different from many matrix/vector settings in existing work, debiasing is not required to establish the asymptotic distribution of estimates or to make statistical inference on low-rank tensors. In fact, due to the widely observed statistical-computational-gap for low-rank tensor estimation, one usually requires stronger conditions than the statistical (or information-theoretic) limit to ensure the computationally feasible estimation is achievable. Surprisingly, such conditions “incidentally” render a feasible low-rank tensor inference without debiasing.

Differentially Private Singular Value Decomposition for Training Support Vector Machines.

Support vector machine (SVM) is an efficient classification method in machine learning. The traditional classification model of SVMs may pose a great threat to personal privacy, when sensitive information is included in the training datasets. Principal component analysis (PCA) can project instances into a low-dimensional subspace while capturing the variance of the matrix A as much as possible. There are two common algorithms that PCA uses to perform the principal component analysis, eigenvalue decomposition (EVD) and singular value decomposition (SVD). The main advantage of SVD compared with EVD is that it does not need to compute the matrix of covariance. This study presents a new differentially private SVD algorithm (DPSVD) to prevent the privacy leak of SVM classifiers. The DPSVD generates a set of private singular vectors that the projected instances in the singular subspace can be directly used to train SVM while not disclosing privacy of the original instances. After proving that the DPSVD satisfies differential privacy in theory, several experiments were carried out. The experimental results confirm that our method achieved higher accuracy and better stability on different real datasets, compared with other existing private PCA algorithms used to train SVM.

Perturbation upper bounds for singular subspaces with a kind of heteroskedastic noise and its application in clustering

Perturbation bounds for singular subspaces have a wide range of applications, including high‐dimensional statistics, machine learning, and applied mathematics. This paper focuses on the perturbation bounds of singular subspaces under a signal matrix interfered by a special heteroscedastic noise matrix in which its same row or column shares similar variance. First of all, we extend homoscedastic results of T. Tony Cai and Anru Zhang (see Rate‐optimal perturbation bounds for singular subspaces with applications to high‐dimensional statistics, The Annals of Statistics, 2018, 46(1), 60–89) to heteroscedastic cases. Then we apply the developed tools to the heteroskedastic clustering model. We find out that our upper bound of clustering misclassification rate is better than the one of T. Tony Cai, Rungang Han, and Anru Zhang (see arXiv:2008.12434, 2020).

Heteroskedastic PCA: Algorithm, optimality, and applications

A general framework for principal component analysis (PCA) in the presence of heteroskedastic noise is introduced. We propose an algorithm called HeteroPCA, which involves iteratively imputing the diagonal entries of the sample covariance matrix to remove estimation bias due to heteroskedasticity. This procedure is computationally efficient and provably optimal under the generalized spiked covariance model. A key technical step is a deterministic robust perturbation analysis on singular subspaces, which can be of independent interest. The effectiveness of the proposed algorithm is demonstrated in a suite of problems in high-dimensional statistics, including singular value decomposition (SVD) under heteroskedastic noise, Poisson PCA, and SVD for heteroskedastic and incomplete data.

Minimax perturbation bounds of the low-rank matrix under Ky Fan norm

<abstract><p>This paper considers the minimax perturbation bounds of the low-rank matrix under Ky Fan norm. We first explore the upper bounds via the best rank-$ r $ approximation $ \hat{A}_r $ of the observation matrix $ \hat{A} $. Next, the lower bounds are established by constructing special matrix groups to show the upper bounds are tight on the low-rank matrix estimation error. In addition, we derive the rate-optimal perturbation bounds for the left and right singular subspaces under Ky Fan norm $ \sin\Theta $ distance. Finally, some simulations have been carried out to support our theories.</p></abstract>

Projected federated averaging with heterogeneous differential privacy

Federated Learning (FL) is a promising framework for multiple clients to learn a joint model without directly sharing the data. In addition to high utility of the joint model, rigorous privacy protection of the data and communication efficiency are important design goals. Many existing efforts achieve rigorous privacy by ensuring differential privacy for intermediate model parameters, however, they assume a uniform privacy parameter for all the clients. In practice, different clients may have different privacy requirements due to varying policies or preferences. In this paper, we focus on explicitly modeling and leveraging the heterogeneous privacy requirements of different clients and study how to optimize utility for the joint model while minimizing communication cost. As differentially private perturbations affect the model utility, a natural idea is to make better use of information submitted by the clients with higher privacy budgets (referred to as "public" clients, and the opposite as "private" clients). The challenge is how to use such information without biasing the joint model. We propose <u>P</u> rojected <u>F</u> ederated <u>A</u> veraging (PFA), which extracts the top singular subspace of the model updates submitted by "public" clients and utilizes them to project the model updates of "private" clients before aggregating them. We then propose communication-efficient PFA+, which allows "private" clients to upload projected model updates instead of original ones. Our experiments verify the utility boost of both algorithms compared to the baseline methods, whereby PFA+ achieves over 99% uplink communication reduction for "private" clients.

A Schatten-q low-rank matrix perturbation analysis via perturbation projection error bound

This paper studies the Schatten-q error of low-rank matrix estimation by singular value decomposition under perturbation. We specifically establish a perturbation bound on the low-rank matrix estimation via a perturbation projection error bound. Then, we establish lower bounds to justify the tightness of the upper bound on the low-rank matrix estimation error. We further develop a user-friendly sinΘ bound for singular subspace perturbation based on the matrix perturbation projection error bound. Finally, we demonstrate the advantage of our results over the ones in the literature by simulation.

Perturbation expansions and error bounds for the truncated singular value decomposition

Truncated singular value decomposition is a reduced version of the singular value decomposition in which only a few largest singular values are retained. This paper presents a novel perturbation analysis for the truncated singular value decomposition for real matrices. First, we describe perturbation expansions for the singular value truncation of order r. We extend perturbation results for the singular subspace decomposition to derive the first-order perturbation expansion of the truncated operator about a matrix with rank greater than or equal to r. Observing that the first-order expansion can be greatly simplified when the matrix has exact rank r, we further show that the singular value truncation admits a simple second-order perturbation expansion about a rank-r matrix. Second, we introduce the first-known error bound on the linear approximation of the truncated singular value decomposition of a perturbed rank-r matrix. Our bound only depends on the least singular value of the unperturbed matrix and the norm of the perturbation matrix. Intriguingly, while the singular subspaces are known to be extremely sensitive to additive noises, the newly established error bound holds universally for perturbations with arbitrary magnitude. Finally, we demonstrate an application of our results to the analysis of the mean squared error associated with the TSVD-based matrix denoising solution.

Singular vector and singular subspace distribution for the matrix denoising model

In this paper, we study the matrix denoising model $Y=S+X$, where $S$ is a low rank deterministic signal matrix and $X$ is a random noise matrix, and both are $M\times n$. In the scenario that $M$ and $n$ are comparably large and the signals are supercritical, we study the fluctuation of the outlier singular vectors of $Y$, under fully general assumptions on the structure of $S$ and the distribution of $X$. More specifically, we derive the limiting distribution of angles between the principal singular vectors of $Y$ and their deterministic counterparts, the singular vectors of $S$. Further, we also derive the distribution of the distance between the subspace spanned by the principal singular vectors of $Y$ and that spanned by the singular vectors of $S$. It turns out that the limiting distributions depend on the structure of the singular vectors of $S$ and the distribution of $X$, and thus they are nonuniversal. Statistical applications of our results to singular vector and singular subspace inferences are also discussed.