Ratio Of Missing Data Research Articles

Time Series Imputation via Integration of Revealed Information Based on the Residual Shortcut Connection

Recovering missing values plays a significant role in time series tasks in practical applications. How to replace the missing data and build the dependency relations from the incomplete sample set is still a challenge. The previous research has found that residual network (ResNet) helps to form a deep network and cope with degradation problem by shortcut connection. Gated recurrent unit (GRU) can improve network model and reduce training parameters by update gate which takes the place of forgetting gate and output gate in long short-term memory (LSTM). Inspired by this finding, we observe that shortcut connection and mean of global revealed information can model the relationship among missing items, the previous and overall revealed information. Hence, we design an imputation network with decay factor for shortcut connection and mean of the global revealed information in GRU, called decay residual mean imputation GRU (DRMI-GRU). We introduce a decay residual mean unit (DRMU), which takes full advantage of the previous and global revealed information to model incomplete time series; and the decay factor is applied to balance the previous long-term dependencies and all non-missing values in the sample set. In addition, a mask unit is designed to check the missing data existing or not. An extensive body of empirical comparisons with other existing imputation algorithms over real-world data and public dataset with different ratio of missing data verifies the performance of our model.

Revisiting Projective Structure from Motion: A Robust and Efficient Incremental Solution.

This paper presents a solution to the Projective Structure from Motion (PSfM) problem able to deal efficiently with missing data, outliers and, for the first time, large scale 3D reconstruction scenarios. By embedding the projective depths into the projective parameters of the points and views, we decrease the number of unknowns to estimate and improve computational speed by optimizing standard linear Least Squares systems instead of homogeneous ones. In order to do so, we show that an extension of the linear constraints from the Generalized Projective Reconstruction Theorem can be transferred to the projective parameters, ensuring also a valid projective reconstruction in the process. We use an incremental approach that, starting from a solvable sub-problem, incrementally adds views and points until completion with a robust, outliers free, procedure. To prevent error accumulation, a refinement based on alternation between new estimations of views and points is used. This can also be done with constrained non-linear optimization. Experiments with simulated data shows that our approach is performing well, both in term of the quality of the reconstruction and the capacity to handle missing data and outliers with a reduced computational time. Finally, results on real datasets shows the ability of the method to be used in medium and large scale 3D reconstruction scenarios with high ratios of missing data (up to 98 percent).

Method for Imputing Missing Data using Online Calibration for Urban Freeway Control

Real-time traffic control systems are widely implemented on roadways around the world as a measure to improve freeway mobility. However, the systems, which rely on data from road-side and on-road sensors and other electronic equipment, continue to suffer from issues related to missing and erroneous data. While many data imputation methods are documented in the related literature, traffic control systems still lack an imputation method that is applicable in practice, accurate in imputation, and simple in computation. In response, this paper puts forth a linear imputation model that considers both temporal traffic trend and spatial detector correlations. To adapt the model to dynamic traffic variations, the imputation method was equipped with an online calibration module. The proposed imputation method was evaluated with field data from two stations on the Whitemud Drive, a busy urban freeway in Edmonton, Alberta, Canada. The proposed model benefited from its time-of-day temporal trend and outperforms the previous model that considers only spatial correlations. Moreover, the online calibration module was effective in improving imputation accuracy. Finally, the sensitivity of imputation performance was analyzed. The results show that the imputation with online calibration is more sensitive to missing data ratios than that with offline calibration. The sensitivity analysis revealed that imputation with online calibration is more suitable for online imputation in traffic control implementations.

Analysis of Differences in Phenology Extracted from the Enhanced Vegetation Index and the Leaf Area Index.

Remote-sensing phenology detection can compensate for deficiencies in field observations and has the advantage of capturing the continuous expression of phenology on a large scale. However, there is some variability in the results of remote-sensing phenology detection derived from different vegetation parameters in satellite time-series data. Since the enhanced vegetation index (EVI) and the leaf area index (LAI) are the most widely used vegetation parameters for remote-sensing phenology extraction, this paper aims to assess the differences in phenological information extracted from EVI and LAI time series and to explore whether either index performs well for all vegetation types on a large scale. To this end, a GLASS (Global Land Surface Satellite Product)-LAI-based phenology product (GLP) was generated using the same algorithm as the MODIS (Moderate Resolution Imaging Spectroradiometer)-EVI phenology product (MLCD) over China from 2001 to 2012. The two phenology products were compared in China for different vegetation types and evaluated using ground observations. The results show that the ratio of missing data is 8.3% for the GLP, which is less than the 22.8% for the MLCD. The differences between the GLP and the MLCD become stronger as the latitude decreases, which also vary among different vegetation types. The start of the growing season (SOS) of the GLP is earlier than that of the MLCD in most vegetation types, and the end of the growing season (EOS) of the GLP is generally later than that of the MLCD. Based on ground observations, it can be suggested that the GLP performs better than the MLCD in evergreen needleleaved forests and croplands, while the MLCD performs better than the GLP in shrublands and grasslands.

Application of a novel hybrid method for spatiotemporal data imputation: A case study of the Minqin County groundwater level

The techniques for data analyses have been widely developed in past years, however, missing data still represent a ubiquitous problem in many scientific fields. In particular, dealing with missing spatiotemporal data presents an enormous challenge. Nonetheless, in recent years, a considerable amount of research has focused on spatiotemporal problems, making spatiotemporal missing data imputation methods increasingly indispensable. In this paper, a novel spatiotemporal hybrid method is proposed to verify and imputed spatiotemporal missing values. This new method, termed SOM-FLSSVM, flexibly combines three advanced techniques: self-organizing feature map (SOM) clustering, the fruit fly optimization algorithm (FOA) and the least squares support vector machine (LSSVM). We employ a cross-validation (CV) procedure and FOA swarm intelligence optimization strategy that can search available parameters and determine the optimal imputation model. The spatiotemporal underground water data for Minqin County, China, were selected to test the reliability and imputation ability of SOM-FLSSVM. We carried out a validation experiment and compared three well-studied models with SOM-FLSSVM using a different missing data ratio from 0.1 to 0.8 in the same data set. The results demonstrate that the new hybrid method performs well in terms of both robustness and accuracy for spatiotemporal missing data.

The ISBSG Software Project Repository: An Analysis from Six Sigma Measurement Perspective for Software Defect Estimation

The International Software Benchmarking Standards Group (ISBSG) provides to researchers and practitioners a repository of software projects’ data that has been used to date mostly for benchmarking and project estimation purposes, but rarely for software defects analysis. Sigma, in statistics, measures how far a process deviates from its goal. Six Sigma focuses on reducing variations within processes, because such variations may lead to an inconsistency in achieving projects’ specifications which represent “defects”, which mean not meeting customers’ satisfaction. Six Sigma provides two methodologies to solve organizations’ problems: “Define-Measure-Analyze-Improve-Control” process cycle (DMAIC) and Design of Six Sigma (DFSS). The DMAIC focuses on improving the existed processes, while the DFSS focuses on redesigning the existing processes and developing new processes. This paper presents an approach to provide an analysis of ISBSG repository based on Six Sigma measurements. It investigates the use of the ISBSG data repository with some of the related Six Sigma measurement aspects, including Sigma defect measurement and software defect estimation. This study presents the dataset preparation consisting of two levels of data preparations, and then analyzed the quality-related data fields in the ISBSG MS-Excel data extract (Release 12 - 2013). It also presents an analysis of the extracted dataset of software projects. This study has found that the ISBSG MS-Excel data extract has a high ratio of missing data within the data fields of “Total Number of Defects” variable, which represents a serious challenge when the ISBSG dataset is being used for software defect estimation.

Multi-task and multi-kernel Gaussian process dynamical systems

In this work, we propose a novel method for rectifying damaged motion sequences in an unsupervised manner. In order to achieve maximal accuracy, the proposed model takes advantage of three key properties of the data: their sequential nature, the redundancy that manifests itself among repetitions of the same task, and the potential of knowledge transfer across different tasks. In order to do so, we formulate a factor model consisting of Gaussian process dynamical systems (GPDS), where each factor corresponds to a single basic pattern in time and is able to represent their sequential nature. Factors collectively form a dictionary of fundamental trajectories shared among all sequences, thus able to capture recurrent patterns within the same or across different tasks. We employ variational inference to learn directly from incomplete sequences and perform maximum a-posteriori (MAP) estimates of the missing values. We have evaluated our model with a number of motion datasets, including robotic and human motion capture data. We have compared our approach to well-established methods in the literature in terms of their reconstruction error and our results indicate significant accuracy improvement across different datasets and missing data ratios. Concluding, we investigate the performance benefits of the multi-task learning scenario and how this improvement relates to the extent of component sharing that takes place.

Missing data imputation of questionnaires by means of genetic algorithms with different fitness functions

This article proposes a new missing data imputation method based on genetic algorithms. The algorithm presented in this paper is a useful tool for the completion of missing data in knowledge and skills tests. This algorithm uses both Bayesian and Akaike’s information criterions as fitness functions and applies them to the classical item response theory models of one, two and three parameters. The results obtained by this new algorithm have been compared with those achieved by means of the Multivariate Imputation by Chained Equations (MICE) algorithm. For all the missing data ratios checked, the average incorrect imputation percentages obtained with the GA algorithm were, statistically, significantly lower than the results obtained with the MICE method. The most favorable frameworks for the use of the algorithm developed in the present research are those questionnaires in which missing answers would be considered as missing completely at random (MCAR). In other words, those questionnaires in which the same questions are present for all the examinees, but not necessarily in the same order.

A Study of Impact on Missing Categorical Data - A Qualitative Review

Objectives: In this study, MissForest algorithm is used to analyze the impact of missing data. Statistical Analysis: For categorical dataset, MissForest package is used to impute various missing values and tested with missing data incrementally, first with 5% missing attributes of the records from the original dataset then with 10% and so on. Findings: The Proportion of Falsely Classified (PFC) is computed for categorical dataset. Since the good performance of MissForest leads to a value close to zero and bad performance to a value around one, the performance measured for the Nursery dataset is quite good. Application: This approach has good effect when the ratio of missing data is low. Missforest algorithm can be used for numerical or categorical data. The results are improved for continuous data.

Multi-view non-negative tensor factorization as relation learning in healthcare data.

Discovering patterns in co-occurrences data between objects and groups of concepts is a useful task in many domains, such as healthcare data analysis, information retrieval, and recommender systems. These relational representations come from objects' behaviors in different views, posing a challenging task of integrating information from these views to uncover the shared latent structures. The problem is further complicated by the high dimension of data and the large ratio of missing data. We propose a new paradigm of learning semantic relations using tensor factorization, by jointly factorizing multi-view tensors and searching for a consistent underlying semantic space across each views. We formulate the idea as an optimization problem and propose efficient optimization algorithms, with a special treatment of missing data as well as high-dimensional data. Experiments results show the potential and effectiveness of our algorithms.

Dynamic mode decomposition for non-uniformly sampled data

We propose an original approach to estimate dynamic mode decomposition (DMD) modes from non-uniformly sampled data. The proposed strategy processes a time-resolved sequence of flow snapshots in three steps. First, a reduced-order modeling of the non-missing data is made by proper orthogonal decomposition to obtain a low-order description of the state space. Second, the missing data are determined with maximum likelihood by coupling a linear dynamical state-space model with the Expectation-Maximization algorithm. Third, the DMD modes are finally estimated on the reconstructed data with a multiple linear regression method called orthonormalized partial least squares regression. This methodology is assessed for the flow past a NACA0012 airfoil at 20° of angle of attack and a Reynolds number of 103. The flow measurements are obtained with time-resolved particle image velocimetry and artificially subsampled at different ratios of missing data. The results show that the proposed method can reproduce the dominant DMD modes and the main structures of the flow fields for 50 and 75 % of missing data.

A Comparative Study of Three Multivariate Short-Term Freeway Traffic Flow Forecasting Methods With Missing Data

Short-term traffic flow forecasting is a critical function in advanced traffic management systems (ATMS) and advanced traveler information systems (ATIS). Accurate forecasting results are useful to indicate future traffic conditions and assist traffic managers in seeking solutions to congestion problems on urban freeways and surface streets. There is new research interest in short-term traffic flow forecasting due to recent developments in intelligent transportation systems (ITS) technologies. Previous research involves technologies in multiple areas, and a significant number of forecasting methods exist in the literature. However, most studies used univariate forecasting methods, and they have limited forecasting abilities when part of the data is missing or erroneous. While the historical average (HA) method is often applied to deal with this issue, the forecasting accuracy cannot be guaranteed. This article makes use of the spatial relationship of traffic flow at nearby locations and builds up two multivariate forecasting approaches: the vector autoregression (VAR) and the general regression neural network (GRNN) based forecasting models. Traffic data collected from U.S. Highway 290 in Houston, TX, were used to test the model performance. Comparison of performances of the three models (HA, VAR, and GRNN) in different missing ratios and forecasting time intervals indicates that the accuracy of the VAR model is more sensitive to the missing ratio, while on average the GRNN model gives more robust and accurate forecasting with missing data, particularly when the missing data ratio is high.

Smooth PARAFAC Decomposition for Tensor Completion

In recent years, low-rank based tensor completion, which is a higher order extension of matrix completion, has received considerable attention. However, the low-rank assumption is not sufficient for the recovery of visual data, such as color and 3D images, when the ratio of missing data is extremely high. In this paper, we consider “smoothness” constraints as well as low-rank approximations and propose an efficient algorithm for performing tensor completion that is particularly powerful regarding visual data. The proposed method admits significant advantages, owing to the integration of smooth PARAFAC decomposition for incomplete tensors and the efficient selection of models in order to minimize the tensor rank. Thus, our proposed method is termed as “smooth PARAFAC tensor completion (SPC).” In order to impose the smoothness constraints, we employ two strategies, total variation (SPC-TV) and quadratic variation (SPC-QV), and invoke the corresponding algorithms for model learning. Extensive experimental evaluations on both synthetic and real-world visual data illustrate the significant improvements of our method, in terms of both prediction performance and efficiency, compared with many state-of-the-art tensor completion methods.

Maximum likelihood estimation of missing data applied to flow reconstruction around NACA profiles

In this paper, we investigate the maximum likelihood estimation for missing data in fluid flows series. The maximum likelihood estimation is provided with the expectation-maximization (EM) algorithm applied to the linear and quadratic proper orthogonal decomposition POD-Galerkin reduced-order models (ROMs) for various sub-samplings of large data sets. The flows around a NACA0012 profile at Reynolds numbers of 103 and angle of incidence of and a NACA0015 profile at Reynolds numbers of 105 and angle of incidence of are first investigated using time-resolved particle image velocimetry measurements and sub-sampled according to different ratios of missing data. The EM algorithm is then applied to the POD ROMs constructed from the sub-sampled data sets. The results show that, depending on the sub-sampling used, the EM algorithm is robust with respect to the Reynolds number and can reproduce the velocity fields and the main structures of the missing flow fields for 50% and 75% of missing data.

Estimating Missing Traffic Volume Using Low Multilinear Rank Tensor Completion

Traffic volume data have been collected and used for various purposes in some aspects of intelligent transportation systems (ITS) applications. However, the unavoidable detector malfunction can cause data to be missing. It is often necessary to develop an effective approach to recover the missing data. In most previous methods, temporal correlation is explored to reconstruct missing traffic volume. In this article, a new missing traffic volume estimation approach based on tensor completion is proposed by exploring traffic spatial–temporal information. The tensor model is utilized to represent traffic volume, which allows for exploring the multicorrelation of traffic volume in spatial and temporal information simultaneously. In order to estimate the missing traffic volume represented by the tensor model, a novel tensor completion algorithm, called low multilinear rank tensor completion, is proposed to reconstruct the missing entries. The proposed approach is evaluated on the PeMS database. Experimental results demonstrate that the proposed method is more effective than the state-of-art methods, especially when the ratio of missing data is high.

Missing Values and Optimal Selection of an Imputation Method and Classification Algorithm to Improve the Accuracy of Ubiquitous Computing Applications

In a ubiquitous environment, high-accuracy data analysis is essential because it affects real-world decision-making. However, in the real world, user-related data from information systems are often missing due to users’ concerns about privacy or lack of obligation to provide complete data. This data incompleteness can impair the accuracy of data analysis using classification algorithms, which can degrade the value of the data. Many studies have attempted to overcome these data incompleteness issues and to improve the quality of data analysis using classification algorithms. The performance of classification algorithms may be affected by the characteristics and patterns of the missing data, such as the ratio of missing data to complete data. We perform a concrete causal analysis of differences in performance of classification algorithms based on various factors. The characteristics of missing values, datasets, and imputation methods are examined. We also propose imputation and classification algorithms appropriate to different datasets and circumstances.

A GS-MPSO-WKNN method for missing data imputation in wireless sensor networks monitoring manufacturing conditions

Wireless sensor networks have been utilized to monitor complex manufacturing processes but missing data from sensors cause problems for data-based applications. In this paper, a missing data estimation algorithm, GS-MPSO-WKNN (Gaussian mutation and simulated annealing-based memetic particle swarm optimization for weighted K nearest neighbours), based on a weighted K nearest neighbour (WKNN) and memetic computing is proposed. The GS-MPSO developed in our previous work is adopted in order to adjust the feature weights for the WKNN. A real world data set from a semiconductor manufacturing process is used to evaluate GS-MPSO-WKNN. Experimental results show that GS-MPSO-WKNN can reach a higher estimation accuracy, and GS-MPSO-WKNN is also robust to a high missing data ratio.

A practical comparison of single and multiple imputation methods to handle complex missing data in air quality datasets

Datasets with missing data ratios ranging from 24% to 4%, corresponding to three air quality monitoring studies, were used to ascertain whether major differences occur when five currently used imputation methods are applied (four single imputation methods and a multiple imputation one). Unrotated and Varimax-rotated factor analyses performed on the imputed datasets were compared. All methods performed similarly, although multiple imputation yielded more disperse imputed values. Main differences occurred when a variable with missing values correlated poorly to the other features and when a variable had relevant loadings in several unrotated factors, which sometimes changed the order of the rotated factors.

A statistical method for characterizing the noise in nonlinearly reconstructed images from undersampled MR data: The POCS example

The projection-onto-convex-sets (POCS) algorithm is a powerful tool for reconstructing high-resolution images from undersampled k-space data. It is a nonlinear iterative method that attempts to estimate values for missing data. The convergence of the algorithm and its other deterministic properties are well established, but relatively little is known about how noise in the source data influences noise in the final reconstructed image. In this paper, we present an experimental treatment of the statistical properties in POCS and investigate 12 stochastic models for its noise distribution beside its nonlinear point spread functions. Statistical results show that as the ratio of the missing k-space data increases, the noise distribution in POCS images is no longer Rayleigh as with conventional linear Fourier reconstruction. Instead, the probability density function for the noise is well approximated by a lognormal distribution. For small missing data ratios, however, the noise remains Rayleigh distributed. Preliminary results show that in the presence of noise, POCS images are often dominated by POCS-enhanced noise rather than POCS-induced artifacts. Implicit in this work is the presentation of a general statistical method that can be used to assess the noise properties in other nonlinear reconstruction algorithms.

Aerobic exercise training during pregnancy increases antioxidant status in nulliparous women: Secondary analysis of a controlled clinical trial

The projection-onto-convex-sets (POCS) algorithm is a powerful tool for reconstructing high-resolution images from undersampled k-space data. It is a nonlinear iterative method that attempts to estimate values for missing data. The convergence of the algorithm and its other deterministic properties are well established, but relatively little is known about how noise in the source data influences noise in the final reconstructed image. In this paper, we present an experimental treatment of the statistical properties in POCS and investigate 12 stochastic models for its noise distribution beside its nonlinear point spread functions. Statistical results show that as the ratio of the missing k-space data increases, the noise distribution in POCS images is no longer Rayleigh as with conventional linear Fourier reconstruction. Instead, the probability density function for the noise is well approximated by a lognormal distribution. For small missing data ratios, however, the noise remains Rayleigh distributed. Preliminary results show that in the presence of noise, POCS images are often dominated by POCS-enhanced noise rather than POCS-induced artifacts. Implicit in this work is the presentation of a general statistical method that can be used to assess the noise properties in other nonlinear reconstruction algorithms.