Incomplete Data Matrices Research Articles

Subspace estimation from unbalanced and incomplete data matrices: ℓ2,∞ statistical guarantees

This paper is concerned with estimating the column space of an unknown low-rank matrix A⋆∈Rd1×d2, given noisy and partial observations of its entries. There is no shortage of scenarios where the observations—while being too noisy to support faithful recovery of the entire matrix—still convey sufficient information to enable reliable estimation of the column space of interest. This is particularly evident and crucial for the highly unbalanced case where the column dimension d2 far exceeds the row dimension d1, which is the focal point of the current paper. We investigate an efficient spectral method, which operates upon the sample Gram matrix with diagonal deletion. While this algorithmic idea has been studied before, we establish new statistical guarantees for this method in terms of both ℓ2 and ℓ2,∞ estimation accuracy, which improve upon prior results if d2 is substantially larger than d1. To illustrate the effectiveness of our findings, we derive matching minimax lower bounds with respect to the noise levels, and develop consequences of our general theory for three applications of practical importance: (1) tensor completion from noisy data, (2) covariance estimation/principal component analysis with missing data and (3) community recovery in bipartite graphs. Our theory leads to improved performance guarantees for all three cases.

Missing Data Imputation Toolbox for MATLAB

Here we introduce a graphical user-friendly interface to deal with missing values called Missing Data Imputation (MDI) Toolbox. This MATLAB toolbox allows imputing missing values, following missing completely at random patterns, exploiting the relationships among variables. In this way, principal component analysis (PCA) models are fitted iteratively to impute the missing data until convergence. Different methods, using PCA internally, are included in the toolbox: trimmed scores regression (TSR), known data regression (KDR), KDR with principal component regression (KDR-PCR), KDR with partial least squares regression (KDR-PLS), projection to the model plane (PMP), iterative algorithm (IA), modified nonlinear iterative partial least squares regression algorithm (NIPALS) and data augmentation (DA). MDI Toolbox presents a general procedure to impute missing data, thus can be used to infer PCA models with missing data, to estimate the covariance structure of incomplete data matrices, or to impute the missing values as a preprocessing step of other methodologies.

Self-organizing maps for imputation of missing data in incomplete data matrices

The problem of incomplete data matrices is repeatedly found in large databases, posing a significant obstacle for an effective treatment of data. This paper examines a self-organizing-map (SOM) based method of data imputation under the concept of distance object per one weight, to predict physicochemical parameters of water samples in a data set where concentrations of different analytes were missed. The method was evaluated according to two different possibilities: (a) including vectors of samples with and without missing data in the training data set and (b) pre-training a SOM for a data set with no missing values and then making imputations for a second data set (prediction set) of samples with missing values. Evaluations were made using a surface water data set of 270 samples from Reconquista River, in Buenos Aires Province, Argentina, by artificially setting a range of 17% to 39% of the data to missing. Results were compared to imputations made through professional criteria. SOMs gave reasonable estimates, with no statistically significant differences from estimates made through professional criteria, proving thus to be a suitable time-saving imputation method.

Comparison of Linear Interpolation Method and Mean Method to Replace the Missing Values in Environmental Data Set

Data collected in air pollution monitoring such as PM10, sulphur dioxide, ozone and carbon monoxide are obtained from automated monitoring stations. These data usually contained missing values due to machine failure, routine maintenance, changes in the siting of monitors and human error. Incomplete datasets can cause bias due to systematic differences between observed and unobserved data. Therefore, it is important to find the best way to estimate these missing values to ensure the quality of data analysed are of high quality. Incomplete data matrices are problematic: incomplete datasets may lead to results that are different from those that would have been obtained from a complete dataset (Hawthorne and Elliott, 2004). There are three major problems that may arise when dealing with incomplete data. First, there is a loss of information and, as a consequence, a loss of efficiency. Second, there are several complications related to data handling, computation and analysis, due to the irregulaties in data structure and the impossibility of using standard software. Third, and more important, there maybe bias due to systematic differences between observed and unobserved data. One approach to solve incomplete data problems is the adoption of imputation techniques (Junninen et al., 2004). Thus, this study compared the performance between linear interpolation method (imputation technique) and substitution of mean value for replacement of missing values in environmental data set.

Two-sample inference for normal mean vectors based on monotone missing data

Inferential procedures for the difference between two multivariate normal mean vectors based on incomplete data matrices with different monotone patterns are developed. Assuming that the population covariance matrices are equal, a pivotal quantity, similar to the Hotelling T 2 statistic, is proposed, and its approximate distribution is derived. Hypothesis testing and confidence estimation of the difference between the mean vectors based on the approximate distribution are outlined. The validity of the approximation is investigated using Monte Carlo simulation. Monte Carlo studies indicate that the approximate method is very satisfactory even for small samples. A multiple comparison procedure is outlined and the proposed methods are illustrated using an example.

Missing data in craniometrics: a simulation study

Craniometric measurements represent a useful tool for studying the differentiation of mammal populations. However, the fragility of skulls often leads to incomplete data matrices. Damaged specimens or incomplete sets of measurements are usually discarded prior to statistical analysis. We assessed the performance of two strategies that avoid elimination of observations: (1) pairwise deletion of missing cells, and (2) estimation of missing data using available measurements. The effect of these distinct approaches on the computation of inter-individual distances and population differentiation analyses were evaluated using craniometric measurements obtained from insular populations of deer micePeromyscus maniculatus (Wagner, 1845). In our simulations, Euclidean distances were greatly altered by pairwise deletion, whereas Gower’s distance coefficient corrected for missing data provided accurate results. Among the different estimation methods compared in this paper, the regression-based approximations weighted by coefficients of determination (r 2) outperformed the competing approaches. We further show that incomplete sets of craniometric measurements can be used to compute distance matrices, provided that an appropriate coefficient is selected. However, the application of estimation procedures provides a flexible approach that allows researchers to analyse incomplete data sets.

Commentary on Bayer's use of incomplete data matrices and weighted qualitative characters in phenetic studies of North American Antennaria: Commentary

Bayer (1987) recognized eight sexual species of Antennaria, A. arornatica Evert, A. corymbosa E., Nels., A. marginata E. L. Greene, A. media E. L. Greene, A. microphylla Rydb., A. racemosa Hook., A. rosulata Rydb., and A. umbrinella Rydb., on the basis of morphometric analysis. We agree that the eight sexual species are morphologically distinct and recognizable (they are generally treated as such), but we disagree with the contention that the species are well defined morphologically by the results obtained from multivariate analysis, specifically, principal components analysis. We contest the analyses of the basic data matrix and submatrices by principal components analysis because the data was (i) known to consist of distinct entities, (ii) for hypothetical OTUs, (iii) incomplete, (iv) both ordinal and nominal, and (v) weighted. Bayer (1987) assumed that principal components analysis was the most appropriate multivariate test for analyzing morphological similarities and differences among the western North American sexual species of Antennaria. We contend that principal components analysis was not the most appropriate multivariate test for this purpose because it is most suitable for the analysis of structure utilizing multivariate observations when no a priori patterns of interrelationships can be discerned (Blackith and Reyment 1971). In truth, any such analysis involves a priori limitation of morphological variation and of potential groups. By selecting 16 specimens of each taxon, except A. arornatica and A. racemosa, Bayer (1987) defined eight groups (species) prior to analysis. Having defined these groups a priori. the data may have been more appropriately analyzed with canonical analysis that maximizes intergroup distances, thereby confirming the a priori groups. In any case, principal components analysis may be an unsatisfactory analytical technique to apply if missing data or qualitative character states are included in the data matrix (Pimentel 1979). The basic data matrix analyzed by Bayer (1987) included qualitative data of the nominal type and was incomplete. Considering these limitations, a more appropriate method of analysis for the purpose of ordination may have been principal coordinate analysis. Although these analytical techniques are similar, the two methods differ in that principal components analysis extracts eigenvalues and eigenvectors from a dispersion matrix of variables or individuals, whereas principal coordinate analysis extracts eigenvalues and eigenvectors from an association matrix of individuals (Gower 1966; Digby

A Comparison of Methods for Treating Incomplete Data in Selection Research

Data were simulated to conform to covariance patterns taken from the personnel selection literature. Two, six, and ten percent of the values were deleted from one of three predictor variables in sample sizes of 50, 100 and 200. Incomplete data matrices were treated by four methods: (a) elimination of cases with incomplete data records; (b) substitution of missing values with the variable mean; (c) replacement of missing values with an estimate obtained from simple regression; and (d) replacement of missing values with an estimate derived from iterated multiple regression. The treated data matrices were subjected to multiple regression analyses, and the resulting regression equations were compared to the equations obtained from the original, complete data. The two regression based estimation procedures provided the most accurate regression equations, followed by the method of inserting means. Discarding cases with incomplete records was the least accurate method. Although the results supported the practice of using covariate information to estimate missing data, the increases in accuracy were minimal under the conditions investigated.

A maximum likelihood solution for the method of successive intervals allowing for unequal stimulus dispersions

A maximum likelihood solution for the method of successive intervals is presented together with an iterative procedure for obtaining estimates for stimulus scale values, boundary scale values, and stimulus dispersions. The method applies without modification to incomplete data matrices. The method was programmed for ILLIAC and a brief description of the program and a numerical illustration are included.