Missing Data Values Research Articles

Missing Values Imputation Based on Iterative Learning

Databases for machine learning and data mining often have missing values. How to develop effective method for missing values imputation is a crucial important problem in the field of machine learning and data mining. In this paper, several methods for dealing with missing values in incomplete data are reviewed, and a new method for missing values imputation based on iterative learning is proposed. The proposed method is based on a basic assumption: There exist cause-effect connections among condition attribute values, and the missing values can be induced from known values. In the process of missing values imputation, a part of missing values are filled in at first and converted to known values, which are used for the next step of missing values imputation. The iterative learning process will go on until an incomplete data is entirely converted to a complete data. The paper also presents an example to illustrate the framework of iterative learning for missing values imputation.

A genetic fuzzy based modeling of effective thermal conductivity for polymer composites

Evaluation of equivalent thermal conductivity ETC of particle reinforced polymer composites PRPCs is a complex process since some of the influencing parameters are associated with uncertainties and ambiguities e.g., dispersion state of filler in the matrix, uniformity of filler particle size and shape, etc. By realizing it, an attempt has been made to model the ETC of 2-phase PRPCs based on a genetic fuzzy approach. The model performance is rigorously tested in three stages to establish its practical applicability: based on experimental data not used in model development cited in literature, new measured thermal conductivities of flexible mould composites and finally by assessing the feasibility of values of missing data in the reported in-complete data set based on the developed model. Estimations of ETC by the proposed model are shown reasonable, even better compare to existing models and suggesting a generic model applicable to a wide range of 2-phase PRPCs.

Dealing with missing data based on data envelopment analysis and halo effect

This research attempts to solve the problem of dealing with missing data via the interface of Data Envelopment Analysis (DEA) and human behavior. Missing data is under continuing discussion in various research fields, especially those highly dependent on data. In practice and research, some necessary data may not be obtained in many cases, for example, procedural factors, lack of needed responses, etc. Thus the question of how to deal with missing data is raised. In this paper, modified DEA models are developed to estimate the appropriate value of missing data in its interval, based on DEA and Inter-dimensional Similarity Halo Effect. The estimated value of missing data is determined by the General Impression of original DEA efficiency. To evaluate the effectiveness of this method, the impact factor is proposed. In addition, the advantages of the proposed approach are illustrated in comparison with previous methods.

Comparison of five iterative imputation methods for multivariate classification

Imputation methods are often used to fill the missing values in an incomplete data set before applying multivariate statistical methods. In this paper, five iterative imputation methods are compared. These include general iterative principal component imputation (GIP), singular value decomposition imputation (SVD), regularized expectation maximization with multiple ridge regression (r-EM), regularized expectation maximization with truncated total least squares (t-EM), and multiple imputation by chained equations (MICE). Two evaluation criteria (covariance change and classification error change) are determined to evaluate imputation performance on one simulated dataset and two published datasets. No single imputation method emerged as the overall best in all cases examined. The r-EM imputation method performs well when the missing proportion is under 20%, judging from results obtained from both real datasets examined. If the percentage of the missing data is above 20%, however, the purpose behind analysis of a dataset should be considered carefully before choosing an imputation method.

Hard c-Means Using Quadratic Penalty-Vector Regularization for Uncertain Data

Clustering is an unsupervised classification technique for data analysis. In general, each datum in real space is transformed into a point in a pattern space to apply clustering methods. Data cannot often be represented by a point, however, because of its uncertainty, e.g., measurement error margin and missing values in data. In this paper, we will introduce quadratic penalty-vector regularization to handle such uncertain data using Hard c-Means (HCM), which is one of the most typical clustering algorithms. We first propose a new clustering algorithm called hard c-means using quadratic penalty-vector regularization for uncertain data (HCMP). Second, we propose sequential extraction hard c-means using quadratic penalty-vector regularization (SHCMP) to handle datasets whose cluster number is unknown. Furthermore, we verify the effectiveness of our proposed algorithms through numerical examples.

Variabilidade espacial e preenchimento de falhas de dados pluviométricos para o estado de Alagoas

A região tropical é marcada por apresentar grande variabilidade quanto à distribuição dos seus regimes pluviométricos, e o conhecimento dessa variabilidade torna-se fundamental para obtenção dos padrões definidores dos regimes hidrológicos e climatológicos desta região. No entanto, a falta de informação quanto à distribuição da precipitação é um sério obstáculo para se compreender e modelar sua variabilidade, surgindo a necessidade de se obter informações para regiões que não apresentam estações de medições ou que apresentem falhas em seu banco de dados através da interpolação. O método consiste em utilizar técnicas geoestatísticas na compreensão da variabilidade espacial e no preenchimento de falhas de dados pluviométricos para o Estado de Alagoas. Para o estudo foram selecionados dados de 63 estações pluviométricas, provenientes da Agência Nacional de Águas (ANA) entre os anos de 1965 a 1980. A utilização da geoestatística, com a análise variográfica, mostrou que as características estruturais estudadas da precipitação estão correlacionadas e apresentam forte dependência espacial. A estimativa da precipitação obtida pelo método de Krigagem Ordinária apresentou resultados satisfatórios, para a distribuição espacial da precipitação, bem como para o preenchimento de falhas.

Interactions between Visual Attention and Episodic Retrieval: Dissociable Contributions of Parietal Regions during Gist-Based False Recognition

The interaction between episodic retrieval and visual attention is relatively unexplored. Given that systems mediating attention and episodic memory appear to be segregated, and perhaps even in competition, it is unclear how visual attention is recruited during episodic retrieval. We investigated the recruitment of visual attention during the suppression of gist-based false recognition, the tendency to falsely recognize items that are similar to previously encountered items. Recruitment of visual attention was associated with activity in the dorsal attention network. The inferior parietal lobule, often implicated in episodic retrieval, tracked veridical retrieval of perceptual detail and showed reduced activity during the engagement of visual attention, consistent with a competitive relationship with the dorsal attention network. These findings suggest that the contribution of the parietal cortex to interactions between visual attention and episodic retrieval entails distinct systems that contribute to different components of the task while also suppressing each other.

An R package suite for microarray meta-analysis in quality control, differentially expressed gene analysis and pathway enrichment detection

With the rapid advances and prevalence of high-throughput genomic technologies, integrating information of multiple relevant genomic studies has brought new challenges. Microarray meta-analysis has become a frequently used tool in biomedical research. Little effort, however, has been made to develop a systematic pipeline and user-friendly software. In this article, we present MetaOmics, a suite of three R packages MetaQC, MetaDE and MetaPath, for quality control, differentially expressed gene identification and enriched pathway detection for microarray meta-analysis. MetaQC provides a quantitative and objective tool to assist study inclusion/exclusion criteria for meta-analysis. MetaDE and MetaPath were developed for candidate marker and pathway detection, which provide choices of marker detection, meta-analysis and pathway analysis methods. The system allows flexible input of experimental data, clinical outcome (case-control, multi-class, continuous or survival) and pathway databases. It allows missing values in experimental data and utilizes multi-core parallel computing for fast implementation. It generates informative summary output and visualization plots, operates on different operation systems and can be expanded to include new algorithms or combine different types of genomic data. This software suite provides a comprehensive tool to conveniently implement and compare various genomic meta-analysis pipelines. http://www.biostat.pitt.edu/bioinfo/software.htm ctseng@pitt.edu Supplementary data are available at Bioinformatics online.

Accurate Tree-based Missing Data Imputation and Data Fusion within the Statistical Learning Paradigm

Framework of this paper is statistical data editing, specifically how to edit or impute missing or contradictory data and how to merge two independent data sets presenting some lack of information. Assuming a missing at random mechanism, this paper provides an accurate tree-based methodology for both missing data imputation and data fusion that is justified within the Statistical Learning Theory of Vapnik. It considers both an incremental variable imputation method to improve computational efficiency as well as boosted trees to gain in prediction accuracy with respect to other methods. As a result, the best approximation of the structural risk (also known as irreducible error) is reached, thus reducing at minimum the generalization (or prediction) error of imputation. Moreover, it is distribution free, it holds independently of the underlying probability law generating missing data values. Performance analysis is discussed considering simulation case studies and real world applications.

Bayesian method for the correlated competitive bidding model

A multivariate competitive bidding model takes into account the correlation among competitors in determination of markup size. However, parameter estimation for the multivariate model is a challenging issue. A simplified, piecemeal style statistical method was proposed for low-dimension problems. However, this method may cause significant estimation errors when applied to complex bidding situations. A refined Bayesian statistical method based on Markov chain Monte Carlo (MCMC) simulation is developed that can be employed in practical bidding problems. To deal with missing values in bid data, a data augmentation technique is integrated in the MCMC process. The proposed Bayesian method is shown through case studies to be robust for complex bidding situations and also insensitive to the selection of the prior models of the correlation matrix. An important feature of the proposed Bayesian method is that it allows a project manager to quantify statistical uncertainties of parameter estimation and their effects on markup decisions. The optimal markup is represented by a posterior distribution which paints a complete picture of the uncertainties involved in the markup size decision.

Social class returns to higher education: comments on a paper by Bukodi and Goldthorpe with a response from the authors

In a recent paper Bukodi E and Goldthorpe JH. (2011) ‘Social class returns to higher education: chances of access to the professional and managerial salariat for men in three British cohorts’. Longitudinal and Life Course Studies, 2, 185-201, 2011 the authors adopt a particular methodological approach to the study of ‘causality’ in life course research. This highlights an important area of contention among social scientists. We have therefore, with the authors’ agreement, taken this as an opportunity to open it up to a wider debate. In this issue of the journal we publish two commentaries, by Clarke, and by Legewie and Solga, that take issue with the Bukodi/Goldthorpe position on causality. The commentaries also question the relationship of the paper’s contribution to the existing economics literature on ‘returns to education’ and the manner in which ‘missing data values’ are handled by the authors. See also a relevant tutorial published in the journal Goldstein H. (2009) ‘Handling attrition and non-response in longitudinal data’. Longitudinal and Life Course Studies, 1, 6372, 2009 that discusses methods for dealing with missing data. We also publish a response by Bukodi and Goldthorpe to these commentaries. We are very grateful to the original authors as well as to those who contributed the commentaries, for taking the time to prepare their carefully argued positions. We hope that this kind of debate will form a regular feature of the journal and help to illuminate important methodological controversies, about which, of course, readers will form their own judgements.

Unsupervised learning of mixture models based on swarm intelligence and neural networks with optimal completion using incomplete data

In this paper, a new algorithm is presented for unsupervised learning of finite mixture models (FMMs) using data set with missing values. This algorithm overcomes the local optima problem of the Expectation-Maximization (EM) algorithm via integrating the EM algorithm with Particle Swarm Optimization (PSO). In addition, the proposed algorithm overcomes the problem of biased estimation due to overlapping clusters in estimating missing values in the input data set by integrating locally-tuned general regression neural networks with Optimal Completion Strategy (OCS). A comparison study shows the superiority of the proposed algorithm over other algorithms commonly used in the literature in unsupervised learning of FMM parameters that result in minimum mis-classification errors when used in clustering incomplete data set that is generated from overlapping clusters and these clusters are largely different in their sizes.

Soft computing based imputation and hybrid data and text mining: The case of predicting the severity of phishing alerts

In this paper, we employ a novel two-stage soft computing approach for data imputation to assess the severity of phishing attacks. The imputation method involves K-means algorithm and multilayer perceptron (MLP) working in tandem. The hybrid is applied to replace the missing values of financial data which is used for predicting the severity of phishing attacks in financial firms. After imputing the missing values, we mine the financial data related to the firms along with the structured form of the textual data using multilayer perceptron (MLP), probabilistic neural network (PNN) and decision trees (DT) separately. Of particular significance is the overall classification accuracy of 81.80%, 82.58%, and 82.19% obtained using MLP, PNN, and DT respectively. It is observed that the present results outperform those of prior research. The overall classification accuracies for the three risk levels of phishing attacks using the classifiers MLP, PNN, and DT are also superior.

A uniquely modern human pattern of endocranial development. Insights from a new cranial reconstruction of the Neandertal newborn from Mezmaiskaya

The globular braincase of modern humans is distinct from all fossil human species, including our closest extinct relatives, the Neandertals. Such adult shape differences must ultimately be rooted in different developmental patterns, but it is unclear at which point during ontogeny these group characteristics emerge.Here we compared internal shape changes of the braincase from birth to adulthood in Neandertals (N = 10), modern humans (N = 62), and chimpanzees (N = 62). Incomplete fossil specimens, including the two Neandertal newborns from Le Moustier 2 and Mezmaiskaya, were reconstructed using reference-based estimation methods. We used 3D geometric morphometrics to statistically compare shapes of virtual endocasts extracted from computed-tomographic scans. Throughout the analysis, we kept track of possible uncertainties due to the missing data values and small fossil sample sizes.We find that some aspects of endocranial development are shared by the three species. However, in the first year of life, modern humans depart from this presumably ancestral pattern of development. Newborn Neandertals and newborn modern humans have elongated braincases, and similar endocranial volumes. During a ‘globularization-phase’ modern human endocasts change to the globular shape that is characteristic for Homo sapiens. This phase of early development is unique to modern humans, and absent from chimpanzees and Neandertals.Our results support the notion that Neandertals and modern humans reach comparable adult brain sizes via different developmental pathways. The differences between these two human groups are most prominent directly after birth, a critical phase for cognitive development.

Comparison of Modeling Approaches for Imputation of Video Detection Data in Intelligent Transportation Systems

Gaps in real-time and archived traffic data are common and can be attributed to several factors, such as sensor failures and data communications interruptions. Regardless of the cause of the missing data, these gaps often must be filled with reliable and accurate estimates before the data can be used for planning, operations, or congestion-mitigation purposes. This research compared different methods for imputing missing values in video detection system data, including historical averages, simple linear regression, multiple linear regression, spatial averages, and Newell's simplified kinematic wave model. The study used the fundamental relationship between speed and flow in filtering the data for quality control. A sensitivity analysis tested the response of different methods to factors such as the size of training data set and time-of-day adjustments to the algorithms. The results indicated that the time of day and volume adjustment factors had a nontrivial impact on the accuracy of the outputs. Despite significant errors in the base data set, the Newell algorithm performed on a par with the other methods in terms of bias and mean absolute percentage, but the more simple factoring methods provided comparable results and were easier to implement.

Multiple Imputation Methods for Multivariate One-Sided Tests with Missing Data

Multivariate one-sided hypotheses testing problems arise frequently in practice. Various tests have been developed. In practice, there are often missing values in multivariate data. In this case, standard testing procedures based on complete data may not be applicable or may perform poorly if the missing data are discarded. In this article, we propose several multiple imputation methods for multivariate one-sided testing problem with missing data. Some theoretical results are presented. The proposed methods are evaluated using simulations. A real data example is presented to illustrate the methods.

Comparative analysis of different techniques for spatial interpolation of rainfall data to create a serially complete monthly time series of precipitation for Sicily, Italy

The availability of good and reliable rainfall data is fundamental for most hydrological analyses and for the design and management of water resources systems. However, in practice, precipitation records often suffer from missing data values mainly due to malfunctioning of raingauge for specific time periods. This is an important issue in practical hydrology because it affects the continuity of rainfall data and ultimately influences the results of hydrologic studies which use rainfall as input. Many methods to estimate missing rainfall data have been proposed in literature and, among these, most are based on spatial interpolation algorithms.In this paper different spatial interpolation algorithms have been evaluated to produce a reasonably good continuous dataset bridging the gaps in the historical series. The algorithms used are deterministic methods such as inverse distance weighting, simple linear regression, multiple regression, geographically weighted regression and artificial neural networks, and geostatistical models such as ordinary kriging and residual ordinary kriging. In some of these methods, the elevation information, provided by a Digital Elevation Model, has been added to improve estimation of missing data. These algorithms have been applied to the mean annual and monthly rainfall data of Sicily (Italy), measured at 247 raingauges.Optimization of different settings of the various interpolation methods has been carried out using a subset of the available rainfall dataset (modeling set) while the remaining subset (validation set) has been used to compare the results obtained by the different algorithms.Validation results indicate that the univariate methods, neglecting the information of elevation, are characterized by the largest errors, which decrease when the elevation is taken into account. The ordinary kriging of residuals from linear regression between precipitation and elevation, which has provided the best performance at annual and monthly scale, has been used to complete the precipitation monthly time series in Sicily.

Evaluation of statistical methods for infilling missing values in high-resolution soil moisture data

Summary Continuous time series of soil moisture data are important for water resource applications including streamflow predictions and weather forecasting. In practice, most soil moisture data have missing values which require infilling to generate complete datasets. This study investigates six methods for infilling missing values for high resolution soil moisture data. This includes multiple linear regression (MLR), weighted Pearson correlation coefficient (WPCC), station relative difference (SRD), soil layer relative difference (SLRD), monthly average replacement (MAR), and merged method. Hourly soil moisture and meteorological data at four different sites, each with nine soil moisture stations located in Halton and Hamilton conservation areas in southern Ontario, Canada were used to evaluate the accuracy levels of the six selected methods. The results showed that the merged method consistently provides the highest evaluation accuracy across all sites. An appealing property of the merged method is its capability to incorporate multiple estimates of missing soil moisture value from other methods. The decreasing order of performance for remaining methods is MAR, SLRD, SRD, MLR and WPCC. The MAR is a simple method but it performed better than complex methods such as MLR and WPCC. Both SRD and SLRD employ the concept of rank stability of soil moisture but SLRD is preferred to SRD – suggesting that similar soil layer at different spatial locations, opposed to temporal persistence of soil moisture in the vertical soil column, is a better predictor of missing soil moisture data. That means, it is more accurate to predict a missing soil moisture at a location using other locations’ soil moisture at a similar depth, than to use soil moisture at a certain depth to predict deeper or shallower soil moisture at the same location.

Multiple Imputation for Missing Values through Conditional Semiparametric Odds Ratio Models

Multiple imputation is a practically useful approach to handling incompletely observed data in statistical analysis. Parameter estimation and inference based on imputed full data have been made easy by Rubin's rule for result combination. However, creating proper imputation that accommodates flexible models for statistical analysis in practice can be very challenging. We propose an imputation framework that uses conditional semiparametric odds ratio models to impute the missing values. The proposed imputation framework is more flexible and robust than the imputation approach based on the normal model. It is a compatible framework in comparison to the approach based on fully conditionally specified models. The proposed algorithms for multiple imputation through the Markov chain Monte Carlo sampling approach can be straightforwardly carried out. Simulation studies demonstrate that the proposed approach performs better than existing, commonly used imputation approaches. The proposed approach is applied to imputing missing values in bone fracture data.

KNN-DTW Based Missing Value Imputation for Microarray Time Series Data

Microarray technology provides an opportunity for scientists to analyze thousands of gene expression profiles simultaneously. However, microarray gene expression data often contain multiple missing expression values due to many reasons. Effective methods for missing value imputation in gene expression data are needed since many algorithms for gene analysis require a complete matrix of gene array values. Several algorithms are proposed to handle this problem, but they have various limitations. In this paper, we develop a novel method to impute missing values in microarray time-series data combining k-nearest neighbor (KNN) and dynamic time warping (DTW). We also analyze and implement several variants of DTW to further improve the efficiency and accuracy of our method. Experimental results show that our method is more accurate compared with existing missing value imputation methods on real microarray time series datasets.