Singular Covariance Research Articles

Fitting a bivariate measurement error model for episodically consumed dietary components.

There has been great public health interest in estimating usual, i.e., long-term average, intake of episodically consumed dietary components that are not consumed daily by everyone, e.g., fish, red meat and whole grains. Short-term measurements of episodically consumed dietary components have zero-inflated skewed distributions. So-called two-part models have been developed for such data in order to correct for measurement error due to within-person variation and to estimate the distribution of usual intake of the dietary component in the univariate case. However, there is arguably much greater public health interest in the usual intake of an episodically consumed dietary component adjusted for energy (caloric) intake, e.g., ounces of whole grains per 1000 kilo-calories, which reflects usual dietary composition and adjusts for different total amounts of caloric intake. Because of this public health interest, it is important to have models to fit such data, and it is important that the model-fitting methods can be applied to all episodically consumed dietary components.We have recently developed a nonlinear mixed effects model (Kipnis, et al., 2010), and have fit it by maximum likelihood using nonlinear mixed effects programs and methodology (the SAS NLMIXED procedure). Maximum likelihood fitting of such a nonlinear mixed model is generally slow because of 3-dimensional adaptive Gaussian quadrature, and there are times when the programs either fail to converge or converge to models with a singular covariance matrix. For these reasons, we develop a Monte-Carlo (MCMC) computation of fitting this model, which allows for both frequentist and Bayesian inference. There are technical challenges to developing this solution because one of the covariance matrices in the model is patterned. Our main application is to the National Institutes of Health (NIH)-AARP Diet and Health Study, where we illustrate our methods for modeling the energy-adjusted usual intake of fish and whole grains. We demonstrate numerically that our methods lead to increased speed of computation, converge to reasonable solutions, and have the flexibility to be used in either a frequentist or a Bayesian manner.

Lossless Linear Transformation of Sensor Data for Distributed Estimation Fusion

In distributed estimation fusion, processed data from each sensor is sent to the fusion center. By taking linear transformation of the raw measurements of each sensor, two optimal distributed fusion algorithms are proposed in this paper. Compared with existing fusion algorithms, they have three nice properties. First, they are optimal in the sense that they are equivalent to the optimal centralized fusion. Second, their communication requirements from each sensor to the fusion center are equal to or less than those of the centralized and most existing distributed fusion algorithms. Third, they do not need the inverses of estimation error covariance matrices, which are assumed to exist in most existing algorithms but can not be guaranteed to exist. So the proposed algorithms can be applied in more cases. Pros and cons of these two new algorithms are analyzed. A possible way to reduce the computational complexity of the new algorithms, an extension to the case of a singular covariance matrix of measurement noise, and an extension to the reduced-rate communication case for some simple systems are also discussed.

Iterative Schur Complement and Multistage Wiener Filtering

The multistage Wiener filter (MWF) is an iterative Schur complement procedure for estimating a scalar or a vector of random variables in the linear minimum mean square error sense. When a complementary orthogonal projector is used as a blocking matrix, intermediate vectors in the development of the MWF have singular covariance matrices, and the rationale of the MWF is undermined. For this situation, we validate the structure of the MWF by showing that it can be developed as an iterative generalized Schur complement procedure which relies on the Moore-Penrose generalized inverse. The validated filter does not require any generalized inverse, and it can be implemented as usual by using scalar Wiener filters when estimating a scalar random variable. Next, we show, under a condition which is commonly met in practice, that the MWF is a least squares reduced rank estimate of the Wiener filter from the Krylov subspace associated with the Wiener-Hopf equation. The MWF is then compared with Brillinger's reduced rank minimum mean square error filter. The two filters differ in the subspaces from which signal estimation is performed and in the order in which projection onto the respective subspaces and Wiener estimation are performed. A forward recursion for the mean square estimation error in each stage of the MWF is provided, and the performance of the MWF is demonstrated by a numerical example.

A Multiplicative Residual Approach To Attitude Kalman Filtering With Unit-Vector Measurements

Using direction vectors of unit length as measurements for attitude estimation in an extended Kalman filter inevitably results in a singular measurement covariance matrix. Singularity of the measurement covariance means no noise is present in one component of the measurement. Unit-vector measurements have no noise in the radial component. Singular measurement covariances can be dealt with by the classic Kalman filter formulation as long as the estimated measurement covariance is non-singular in the same direction. Unit-vector measurements violate this condition because both the true measurement and the estimated measurement have perfectly known lengths. Minimum variance estimation for the unit-vector attitude Kalman filter is studied in this work. An optimal multiplicative residual approach is presented. The proposed approach is compared with the classic additive residual attitude Kalman filter.

Generalized Mahalanobis depth in the reproducing kernel Hilbert space

In this paper, Mahalanobis depth (MHD) in the Reproducing Kernel Hilbert Space (RKHS) is proposed. First, we extend the notion of MHD to a generalized version, i.e., the generalized MHD (GHMD), to make it suitable for the small sample with singular covariance matrix. We prove that GMHD is consistent with MHD when the sample has a full-rank covariance matrix. Second, we further extend GMHD to RKHS, i.e, the kernel mapped GMHD (kmGMHD), and discuss its main properties. Numeric results show that kmGMHD can give a better depth interpretation for the sample with special shape, such as a non-convex sample set. Our proposed kmGMHD can be potentially used as a robust tool for outliers detection and data classification. In addition, we also discuss the influence of parameters on the shape of the central regions.

Equality of BLUEs or BLUPs under two linear models using stochastic restrictions

In this paper, we consider mixed linear models, possibly with singular covariance matrices, by supplementing a particular fixed effects model with appropriate stochastic restrictions. We show that all representations of the best linear unbiased estimator (BLUE) and best linear unbiased predictor (BLUP) can be obtained through the augmented model including stochastic restrictions. Using this approach, we consider two mixed linear models, \({\fancyscript {M}_1}\) and \({\fancyscript {M}_2}\) , say, which have different covariance matrices. We give necessary and sufficient conditions that the BLUP and/or BLUE under the the model \({\fancyscript {M}_{1}}\) continue to be BLUP and/or BLUE also under the model \({\fancyscript {M}_{2}}\) .

Parameter Uncertainty in Portfolio Selection: Shrinking the Inverse Covariance Matrix

The estimation of the inverse covariance matrix plays a crucial role in optimal portfolio choice. We propose a new estimation framework that focuses on enhancing portfolio performance. The framework applies the statistical methodology of shrinkage directly to the inverse covariance matrix using two non-parametric methods. The first minimises the out-of-sample portfolio variance while the second aims to increase out-of-sample risk-adjusted returns. We apply the resulting estimators to compute the minimum variance portfolio weights and obtain a set of new portfolio strategies. These strategies have an intuitive form which allows us to extend our framework to account for short-sale constraints, high transaction costs and singular covariance matrices. A comparative empirical analysis against several strategies from the literature shows that the new strategies generally offer higher risk-adjusted returns and lower levels of risk.

Markov-Chain Approximations of Vector Autoregressions: Application of General Multivariate-Normal Integration Techniques

Discrete Markov chains can be useful to approximate vector autoregressive processes for economists doing computational work. One such approximation method first presented by Tauchen (1986) operates under the general theoretical assumption of a transformed VAR with diagonal covariance structure for the process error term. We demonstrate one simple method of more conveniently treating this approximation problem in practice using readily available multivariate-normal integration techniques to allow for arbitrary positive-semidefinite covariance structures. Examples are provided using processes with non-diagonal and singular non-diagonal error covariances.

Cluster-based visualisation with scatter matrices

The trace of the scatter matrix, which measures separation between population cohorts, is shown to be strictly preserved by sphering the data followed by a projection onto the space of population means. This result suggests using the space of means as a basis to calculate well-separating lower-dimensional projections of the data, derived from the scatter matrix in the projective space. In particular, it defines an approximation to the canonical decomposition of the scatter matrix that applies for singular covariance matrices. The method is illustrated with reference to k-means clusters in data sets from bioinformatics and marketing.

Structural equation modeling with near singular covariance matrices

Conventional structural equation modeling involves fitting a structural model to the sample covariance matrix S . Due to collinearity or small samples with practical data, nonconvergences often occur in the estimation process. For a small constant a , this paper proposes to fit the structural model to the covariance matrix S a = S + a I . When treating S a as the sample covariance matrix in the maximum likelihood (ML) procedure, consistent parameter estimates are still obtained. The asymptotic distributions of the parameter estimates and the corresponding likelihood ratio statistic are studied and compared to those by the conventional ML. Two rescaled statistics for the overall model evaluation with modeling S a are constructed. Empirical results imply that the estimates from modeling S a are more efficient than those of fitting the structural model to S even when data are normally distributed. Simulations and real data examples indicate that modeling S a allows us to evaluate the overall model structure even when S is literally singular. Implications of modeling S a in a broader context are discussed.

Heteroscedastic Low-Rank Matrix Approximation by the Wiberg Algorithm

Low-rank matrix approximation has applications in many fields, such as 2-D filter design and 3-D reconstruction from an image sequence. In this paper, one issue with low-rank matrix approximation is investigated: heteroscedastic noise. In most of previous research, the covariance matrix of the heteroscedastic noise is assumed to be positive definite. This requirement restricts the usefulness of results derived from such research. In this paper, we extend the Wiberg algorithm, which originally deals with the missing data problem with low-rank approximation, to the cases, where the heteroscedastic noise has a singular covariance matrix. Experiments show that the proposed Wiberg algorithm converges much faster than the bilinear approach, and consequently avoids many nonconvergent cases in the bilinear approach. Experiments also show that, to some extent, the Wiberg algorithm can tolerate outliers and is not sensitive to parameter variation.

Attitude Filtering on SO(3)

A new method is presented for the simultaneous estimation of the attitude of a spacecraft and an N-vector of bias parameters. This method uses a probability distribution function defined on the Cartesian product of SO(3), the group of rotation matrices, and the Euclidean space {xc391-1} The Fokker-Planck equation propagates the probability distribution function between measurements, and Bayes’ formula incorporates measurement update information. This approach avoids all the issues of singular attitude representations or singular covariance matrices encountered in extended Kalman filters. In addition, the filter has a consistent initialization for a completely unknown initial attitude, owing to the fact that SO(3) is a compact space.

Decomposition of Toeplitz Matrices via Convex Optimization

We point out that autocovariance functions of moving average processes of any given order m can be characterized via a linear matrix inequality (LMI). This LMI-condition can be used to decompose any Toeplitz autococovariance matrix into a sum of a singular Toeplitz covariance plus the autocovariance matrix of a moving average process of order m and of maximal variance. The decomposition is unique and subsumes the Pisarenko harmonic decomposition that corresponds to m=0. It can be used to account for mutual couplings between elements in linear antenna arrays or identify colored noise consistent with the covariance data. The same LMI-condition leads to an efficient computation of the least order of a MA-spectrum that agrees with covariance moments

A modification of canonical variates analysis to handle highly collinear multivariate data

AbstractA modification of the standard Canonical Variates Analysis (CVA) method to cope with collinear high‐dimensional data is developed. The method utilizes Partial Least Squares regression as an engine for solving an eigenvector problem involving singular covariance matrices. Three data sets are analyzed to demonstrate the properties of the method: a two‐group problem with near infrared spectroscopic data consisting of 60 samples and 376 variables, a multi‐group problem with fluorescence spectroscopic data (1023 variables) consisting of 83 samples from six groups and a three‐group problem with physical‐chemical data (10 variables) consisting of 41 samples from three groups. It is demonstrated that the modified CVA method forces the discriminative information into the first canonical variates as expected. The weight vectors found in the modified CVA method possess the same properties as weight vectors of the standard CVA method. By combination of the suggested method with, for example, Linear Discriminant Analysis (LDA) as a classifier, an operational tool for classification and discrimination of collinear data is obtained. Copyright © 2007 John Wiley & Sons, Ltd.

INCOMPRESSIBLE FLUIDS

We propose a model for random forces in a turbulent incompressible fluid by balancing the energy gain from fluctuations against dissipation by viscosity. This leads to a more singular covariance distribution for the random forces than is ordinarily allowed. We then propose regularization of the fluid system by matrix models. A formula for entropy of a two dimensional fluid is derived and then a vorticity profile of a hurricane that maximizes entropy. A regularization of three dimensional incompressible fluid flow using quantum groups is also proposed.

The geometry of linear separability in data sets

We study the geometry of datasets, using an extension of the Fisher linear discriminant to the case of singular covariance, and a new regularization procedure. A dataset is called linearly separable if its different clusters can be reliably separated by a linear hyperplane. We propose a measure of linear separability, easily computed as an angle that arises naturally in our analysis. This angle of separability assumes values between 0 and π/2, with high [resp. low] values corresponding to datasets that are linearly separable, resp. inseparable.

Signal behavior of adaptive filtering algorithms in a nonstationary environment with singular data covariance matrix

The paper analyzes the signal behavior of adaptive filtering algorithms when the target weights of the adaptive filter are time varying and the covariance matrix of the filter input is singular. The signal behavior is evaluated in terms of moments of the excess output error of the filter. Two algorithms are considered: the LMS algorithm and the sign algorithm. The analysis is done in the context of adaptive plant identification. The plant parameters vary according to a random walk model. The plant input, plant noise, and plant parameters are assumed mutually independent. Under these assumptions, it is found that the signal behavior of the algorithms is the same as the signal behavior in the case with positive definite input covariance matrix.

$V$-invariant methods, generalised least squares problems, and the Kalman filter

V-invariant methods for the generalised least squares problem extend the techniques based on orthogonal factorization for ordinary least squares to problems with multiscaled, even singular covariances. These methods are summarised briefly here, and the ability to handle multiple scales indicated. An application to a class of Kalman filter problems derived from generalised smoothing splines is considered. Evidence of severe illconditioning of the covariance matrices is demonstrated in several examples. This suggests that this is an appropriate application for the V-invariant techniques.

The General Multivariate Gauss—Markov Model of the Incomplete Block Design

AbstractThe aim of the paper is to generalize testing and estimation for the multivariate standard incomplete block model (Rao & Mitra, 1971a) to the general multivariate Gauss—Markov incomplete block model with singular covariance matrix. The results of this paper can be applied to particular cases of the multivariate Gauss—Markov incomplete block model, including the Zyskind—Martin model.

Saddlepoint Expansions in Linear Regression

In this paper sums of independent but not identically distributed m-dimensional vectors are considered. The summands are generated by a random vector multiplied by a deterministic weight matrix which results in a singular covariance matrix. Under general conditions, given separately for the weight matrix and the random vector, saddlepoint approximations to the distribution of the sum are derived. The results are applied to the least squares estimator, the residual sum of squares, and to an F-statistic in linear regression.