Empirical Covariance Operators Research Articles

Regular variation in Hilbert spaces and principal component analysis for functional extremes

Motivated by the increasing availability of data of functional nature, we develop a general probabilistic and statistical framework for extremes of regularly varying random elements X in L2[0,1]. We place ourselves in a Peaks-Over-Threshold framework where a functional extreme is defined as an observation X whose L2-norm ‖X‖ is comparatively large. Our goal is to propose a dimension reduction framework resulting into finite dimensional projections for such extreme observations. Our contribution is double. First, we investigate the notion of Regular Variation for random quantities valued in a general separable Hilbert space, for which we propose a novel concrete characterization involving solely stochastic convergence of real-valued random variables. Second, we propose a notion of functional Principal Component Analysis (PCA) accounting for the principal ‘directions’ of functional extremes. We investigate the statistical properties of the empirical covariance operator of the angular component of extreme functions, by upper-bounding the Hilbert–Schmidt norm of the estimation error for finite sample sizes. Numerical experiments with simulated and real data illustrate this work.

Relative perturbation bounds with applications to empirical covariance operators

The goal of this paper is to establish relative perturbation bounds, tailored for empirical covariance operators. Our main results are expansions for empirical eigenvalues and spectral projectors, leading to concentration inequalities and limit theorems. One of the key ingredients is a specific separation measure for population eigenvalues, which we call the relative rank, giving rise to a sharp invariance principle in terms of limit theorems, concentration inequalities and inconsistency results. Our framework is very general, requiring only p>4 moments and allows for a huge variety of dependence structures.

Nonasymptotic upper bounds for the reconstruction error of PCA

We analyse the reconstruction error of principal component analysis (PCA) and prove nonasymptotic upper bounds for the corresponding excess risk. These bounds unify and improve existing upper bounds from the literature. In particular, they give oracle inequalities under mild eigenvalue conditions. The bounds reveal that the excess risk differs significantly from usually considered subspace distances based on canonical angles. Our approach relies on the analysis of empirical spectral projectors combined with concentration inequalities for weighted empirical covariance operators and empirical eigenvalues.

Randomized estimation of functional covariance operator via subsampling

Covariance operators are fundamental concepts and modelling tools for many functional data analysis methods, such as functional principal component analysis. However, the empirical (or estimated) covariance operator becomes too costly to compute when the functional dataset gets big. This paper studies a randomized algorithm for covariance operator estimation. The algorithm works by sampling and rescaling observations from the large functional data collection to form a sketch of much smaller size and performs computation on the sketch to obtain the subsampled empirical covariance operator. The proposed algorithm is theoretically justified via nonasymptotic bounds between the subsampled and the full‐sample empirical covariance operator in terms of the Hilbert‐Schmidt norm and the operator norm. It is shown that the optimal sampling probability that minimizes the expected squared Hilbert‐Schmidt norm of the subsampling error is determined by the norm of each function. Simulated and real data examples are used to illustrate the effectiveness of the proposed algorithm.

Perturbation bounds for eigenspaces under a relative gap condition

A basic problem in operator theory is to estimate how a small perturbation affects the eigenspaces of a self-adjoint compact operator. In this paper, we prove upper bounds for the subspace distance, tailored for relative perturbations. As a main example, we consider the empirical covariance operator and show that a sharp bound can be achieved under a relative gap condition. The proof is based on a novel contraction phenomenon, contrasting previous spectral perturbation approaches.

Hybrid regularisation and the (in)admissibility of ridge regression in infinite dimensional Hilbert spaces

We consider the problem of estimating the slope function in a functional regression with a scalar response and a functional covariate. This central problem of functional data analysis is well known to be ill-posed, thus requiring a regularised estimation procedure. The two most commonly used approaches are based on spectral truncation or Tikhonov regularisation of the empirical covariance operator. In principle, Tikhonov regularisation is the more canonical choice. Compared to spectral truncation, it is robust to eigenvalue ties, while it attains the optimal minimax rate of convergence in the mean squared sense, and not just in a concentration probability sense. In this paper, we show that, surprisingly, one can strictly improve upon the performance of the Tikhonov estimator in finite samples by means of a linear estimator, while retaining its stability and asymptotic properties by combining it with a form of spectral truncation. Specifically, we construct an estimator that additively decomposes the functional covariate by projecting it onto two orthogonal subspaces defined via functional PCA; it then applies Tikhonov regularisation to the one component, while leaving the other component unregularised. We prove that when the covariate is Gaussian, this hybrid estimator uniformly improves upon the MSE of the Tikhonov estimator in a non-asymptotic sense, effectively rendering it inadmissible. This domination is shown to also persist under discrete observation of the covariate function. The hybrid estimator is linear, straightforward to construct in practice, and with no computational overhead relative to the standard regularisation methods. By means of simulation, it is shown to furnish sizeable gains even for modest sample sizes.

Geometric Sensitivity Measures for Bayesian Nonparametric Density Estimation Models

We propose a geometric framework to assess global sensitivity in Bayesian nonparametric models for density estimation. We study sensitivity of nonparametric Bayesian models for density estimation, based on Dirichlet-type priors, to perturbations of either the precision parameter or the base probability measure. To quantify the different effects of the perturbations of the parameters and hyperparameters in these models on the posterior, we define three geometrically-motivated global sensitivity measures based on geodesic paths and distances computed under the nonparametric Fisher-Rao Riemannian metric on the space of densities, applied to posterior samples of densities: (1) the Fisher-Rao distance between density averages of posterior samples, (2) the log-ratio of Karcher variances of posterior samples, and (3) the norm of the difference of scaled cumulative eigenvalues of empirical covariance operators obtained from posterior samples. We validate our approach using multiple simulation studies, and consider the problem of sensitivity analysis for Bayesian density estimation models in the context of three real datasets that have previously been studied.

A class of optimal estimators for the covariance operator in reproducing kernel Hilbert spaces

The covariance operator plays an important role in modern statistical methods and is critical for inference. It is most often estimated by the empirical covariance operator. In spite of its simple and appealing properties, however, this estimator can be improved by a class of shrinkage operators. In this paper, we study shrinkage estimation of the covariance operator in reproducing kernel Hilbert spaces. A data-driven shrinkage estimator enjoying desirable theoretical and computational properties is proposed. The procedure is easily implemented and its numerical performance is investigated through simulations. In finite samples, the estimator outperforms the empirical covariance operator, especially when the data dimension is much larger than the sample size. We also show that the rate of convergence in Hilbert–Schmidt norm is of the order n−1∕2. Furthermore, we establish the minimax optimal rate of convergence over suitable classes of probability measures and demonstrate that these shrinkage operators are all minimax rate-optimal.

Optimal rate for covariance operator estimators of functional autoregressive processes with random coefficients

We improve a result of Allam and Mourid (2014) by deriving the optimal n rate for the empirical covariance operators of a Hilbert-valued autoregressive process with random coefficients. Our approach is based on a suitable autoregressive representation of a sequence of covariance operators related to the model, which leads to a decomposition with Hilbert-valued martingale differences. Using large deviation inequalities for Hilbert-valued martingale differences, we then establish exponential bounds and derive the almost sure convergence of the empirical covariance operators in the Hilbert–Schmidt norm, achieving the parametric rate n up to a ln(n) factor in the bounded process case.

Asymptotic theory of multiple-set linear canonical analysis

This paper deals with asymptotics for multiple-set linear canonical analysis (MSLCA). A definition of this analysis, that adapts the classical one to the context of Euclidean random variables, is given and properties of the related canonical coefficients are derived. Then, estimators of the MSLCA’s elements, based on empirical covariance operators, are proposed and asymptotics for these estimators is obtained. More precisely, we prove their consistency and we obtain asymptotic normality for the estimator of the operator that gives MSLCA, and also for the estimator of the vector of canonical coefficients. These results are then used to obtain a test for mutual non-correlation between the involved Euclidean random variables.

Testing for independence between functional time series

Frequently econometricians are interested in verifying a relationship between two or more time series. Such analysis is typically carried out by causality and/or independence tests which have been well studied when the data is univariate or multivariate. Modern data though is increasingly of a high dimensional or functional nature for which finite dimensional methods are not suitable. In the present paper we develop methodology to check the assumption that data obtained from two functional time series are independent. Our procedure is based on the norms of empirical cross covariance operators and is asymptotically validated when the underlying populations are assumed to be in a class of weakly dependent random functions which include the functional ARMA, ARCH and GARCH processes.

TESTING EQUALITY OF MEANS WHEN THE OBSERVATIONS ARE FROM FUNCTIONAL TIME SERIES

There are numerous examples of functional data in areas ranging from earth science to finance where the problem of interest is to compare several functional populations. In many instances, the observations are obtained consecutively in time, and thus, the classical assumption of independence within each population may not be valid. In this article, we derive a new, asymptotically justified method to test the hypothesis that the mean curves of multiple functional populations are the same. The test statistic is constructed from the coefficient vectors obtained by projecting the functional observations into a finite dimensional space. Asymptotics are established when the observations are considered to be from stationary functional time series. Although the limit results hold for projections into arbitrary finite dimensional spaces, we show that higher power is achieved by projecting onto the principle components of empirical covariance operators that diverge under the alternative. Our method is further illustrated by a simulation study as well as an application to electricity demand data.

Consistency of the mean and the principal components of spatially distributed functional data

This paper develops a framework for the estimation of the functional mean and the functional principal components when the functions form a random field. More specifically, the data we study consist of curves $X(\mathbf{s}_k;t),t\in[0,T]$, observed at spatial points $\mathbf{s}_1,\mathbf{s}_2,\ldots,\mathbf{s}_N$. We establish conditions for the sample average (in space) of the $X(\mathbf{s}_k)$ to be a consistent estimator of the population mean function, and for the usual empirical covariance operator to be a consistent estimator of the population covariance operator. These conditions involve an interplay of the assumptions on an appropriately defined dependence between the functions $X(\mathbf{s}_k)$ and the assumptions on the spatial distribution of the points $\mathbf{s}_k$. The rates of convergence may be the same as for i.i.d. functional samples, but generally depend on the strength of dependence and appropriately quantified distances between the points $\mathbf{s}_k$. We also formulate conditions for the lack of consistency.

Second-Order Comparison of Gaussian Random Functions and the Geometry of DNA Minicircles

Given two samples of continuous zero-mean iid Gaussian processes on [0,1], we consider the problem of testing whether they share the same covariance structure. Our study is motivated by the problem of determining whether the mechanical properties of short strands of DNA are significantly affected by their base-pair sequence; though expected to be true, had so far not been observed in three-dimensional electron microscopy data. The testing problem is seen to involve aspects of ill-posed inverse problems and a test based on a Karhunen–Loève approximation of the Hilbert–Schmidt distance of the empirical covariance operators is proposed and investigated. When applied to a dataset of DNA minicircles obtained through the electron microscope, our test seems to suggest potential sequence effects on DNA shape. Supplemental material available online.

A sufficient condition for the CLT in the space of nuclear operators—Application to covariance of random functions

We give a sufficient condition for the CLT to hold in the space of trace class (nuclear) operators. This condition turns out to be adapted to the asymptotic study of empirical covariance operators of Hilbert valued random variables.

Weak convergence for the covariance operators of a Hilbertian linear process

Let X t =∑ k=−∞ +∞ a k ( ε t− k ) be a linear process with values in a Hilbert space H. The H valued r.v. ε k are i.i.d. centered, the a k 's are linear operators. We prove a central limit theorem for the vector of empirical covariance operators of the random variables X t at orders 0 to h∈ N in the space of Hilbert–Schmidt operators. Statistical applications are given in the area of principal component analysis for vector dependent random curves.

Estimation of mean and covariance operator of autoregressive processes in Banach spaces

The autoregressive model in a Banach space (ARB) contains many continuous time processes used in practice, for example, processes that satisfy linear stochastic differential equations of order k, a very particular case being the Ornstein–Uhlenbeck process. In this paper we study empirical estimators for ARB processes. In particular we show that, under some regularity conditions, the empirical mean is asymptotically optimal with respect to a.s. convergence and convergence of order 2. Limit in distribution and the law of the iterated logarithm are also presented. Concerning the empirical covariance operator we note that, if (X n, n ∈ Z) is ARB then (X n ⊗ X n, n ∈ Z) is AR in a suitable space of linear operators. This fact allows us to interpret the empirical covariance operator as a sample mean of an AR and to derive similar results for it.

Nonparametric estimation of smoothed principal components analysis of sampled noisy functions

This study deals with the simultaneous nonparametric estimations of n curves or observations of a random process corrupted by noise in which sample paths belong to a finite dimension functional subspace. The estimation, by means of B-splines, leads to a new kind of functional principal components analysis. Asymptotic rates of convergence are given for the mean and the eigenelements of the empirical covariance operator. Heuristic arguments show that a well chosen smoothing parameter may improve the estimation of the subspace which contains the sample path of the process. Finally, simulations suggest that the estimation method studied here is advantageous when there are a small number of design points.

Normalité asymptotique de l'estimateur empirique de l'opérateur d'autocorrélation d'un processus ARH(1)

We consider the ARH(1) model defined by X t =ρ( X t−1 + ε t , where p is a linear operator on a Hilbert space; X t and ε t are Hilbert space valued random variables ((ε t ) t ∈N is a white noise). We derive a central limit theorem for an estimator of the autocorrelation operator p based on the empirical covariance operators of the process and on the projection of the data on a subspace whose dimension increases with the number of observations.

Représentation autorégressive de l'opérateur de covariance empirique d'un ARH(1). Applications

Given an order 1 autoregressive process with values in a separable Hilbert space we show that its empirical covariance operator, say Cn, is the empirical mean of an autoregressive process which takes its values in the space of Hilbert-Schmidt operators with a noise which is an Hilbertian martingale difference. This allows to obtain precise results concerning the asymptotic behaviour of C n and its eigenvalues.