Missing Value Imputation Techniques Research Articles

Revisiting COVID-19’s Impact on Cycling: An Examination of Bicycle Count Data in Montréal

During the earlier stages of the COVID-19 pandemic, numerous studies used bicycle count data to understand the pandemic’s impact on urban cycling. However, given the context and timeliness of such studies, correcting for periods of missing data and ensuring data accuracy were not always possible. In this paper, we examine the quality of bicycle count data, using Montreal as a case study, and propose methodological improvements. By employing a missing value imputation technique and excluding sensors potentially affected by exogenous factors, we hope to provide cities with a more precise assessment of cycling trends.

Smartic: A smart tool for Big Data analytics and IoT

The Internet of Things (IoT) is leading the physical and digital world of technology to converge. Real-time and massive scale connections produce a large amount of versatile data, where Big Data comes into the picture. Big Data refers to large, diverse sets of information with dimensions that go beyond the capabilities of widely used database management systems, or standard data processing software tools to manage within a given limit. Almost every big dataset is dirty and may contain missing data, mistyping, inaccuracies, and many more issues that impact Big Data analytics performances. One of the biggest challenges in Big Data analytics is to discover and repair dirty data; failure to do this can lead to inaccurate analytics results and unpredictable conclusions. Different imputation methods were employed in the experimentation with various missing value imputation techniques, and the performances of machine learning (ML) models were compared. A hybrid model that integrates ML and sample-based statistical techniques for missing value imputation is being proposed. Furthermore, the continuation involved the dataset with the best missing value imputation, chosen based on ML model performance for subsequent feature engineering and hyperparameter tuning. K-means clustering and principal component analysis were applied in our study. Accuracy, the evaluated outcome, improved dramatically and proved that the XGBoost model gives very high accuracy at around 0.125 root mean squared logarithmic error (RMSLE). To overcome overfitting, K-fold cross-validation was implemented.

Partial Discharge Localization through k-NN and SVM

Power transformers are essential for the distribution and transmission of electricity, but they are prone to degradation due to faults early on. Partial Discharge (PD) is the most significant pointer of insulation breakdown in high-voltage apparatus. Dissolved Gas Analysis (DGA) is a commonly used technique for detecting and diagnosing PD. However, DGA data often contain missing values, which can significantly affect the accuracy of PD diagnosis. To mitigate the issues of missing values, this paper proposes using the k-Nearest Neighbors (kNN) technique to impute the missing values in the dataset. Further, it combines kNN with a Support Vector Machine (SVM) to detect the possibility of a PD source in the high-voltage apparatus. The approach was evaluated on a real-world DGA dataset and achieved high classification performance and discriminatory power for distinguishing between PD and non-PD instances. The effectiveness of the missing value imputation technique was evaluated, and the proposed approach demonstrated improved accuracy and precision compared to methods without imputation. The proposed approach offers a current solution for PD analysis in power transformers using DGA data with missing values.

Exploring explicit and implicit graph learning for multivariate time series imputation

Multivariate time series inherently contain missing values due to various issues, including incorrect data entry, broken equipment, and package loss during data transferring. The successful completion of time series data analysis tasks heavily relies on the essential task of imputing missing values. Inter-variable relationships in time series are typically overlooked by missing value imputation techniques. Although some graph-based algorithms can capture these relationships, the design of graph structures is commonly handcrafted and dataset-centric. We introduce a novel Explicit and Implicit Graph Recurrent Network (EIGRN) for multivariate time series imputation that integrates graph and recurrent neural networks to capture variable and time dependencies together. This proficiency is achieved by effectively integrating external data sources such as domain knowledge and the implicit relationships among nodes. In order to make our approach more applicable to datasets with larger numbers of missing values, we additionally discuss the model’s performance for various missing value ratios. Our comprehensive experiments on real-world datasets show that our model outperforms state-of-the-art baselines in different industrial fields.

Outlier Elimination Technique Using Deletion‐Imputation Iteration for Fault‐Prone Module Detection

In software development, improving the efficiency of a testing process is important to ensure reliability during a limited development period. One approach to improving the testing process is identifying modules that are likely to contain faults (fault‐prone modules) and allocate effort toward resolving them. For this purpose, many fault‐prone module detection models have been proposed in previous studies. However, an appropriate fault‐prone module detection model cannot be constructed if outliers, such as modules with a significantly large number of source code lines and branches, but no faults, are included. In this study, we propose a new outlier elimination technique that creates missing values (deleted values) to complete the data artificially and applies a missing value imputation technique using a regression approach. If the imputed value differs significantly from the actual (recorded) value, the proposed technique treats the values as outliers. We name the proposed technique Outlier Elimination Technique using Deletion‐Imputation Iteration (OEdii), and its performance is verified experimentally. In general, our experimental results are more accurate than the previous outlier elimination techniques in the area under the ROC curve. © 2023 Institute of Electrical Engineer of Japan and Wiley Periodicals LLC.

Comparative Study of Missing Value Imputation Techniques on E-Commerce Product Ratings

Comparative Study of Missing Value Imputation Techniques on E-Commerce Product Ratings

Fuzzy-based missing value imputation technique for air pollution data

Fuzzy-based missing value imputation technique for air pollution data

Smartic: A smart tool for Big Data analytics and IoT.

The Internet of Things (IoT) is leading the physical and digital world of technology to converge. Real-time and massive scale connections produce a large amount of versatile data, where Big Data comes into the picture. Big Data refers to large, diverse sets of information with dimensions that go beyond the capabilities of widely used database management systems, or standard data processing software tools to manage within a given limit. Almost every big dataset is dirty and may contain missing data, mistyping, inaccuracies, and many more issues that impact Big Data analytics performances. One of the biggest challenges in Big Data analytics is to discover and repair dirty data; failure to do this can lead to inaccurate analytics results and unpredictable conclusions. Different imputation methods were employed in the experimentation with various missing value imputation techniques, and the performances of machine learning (ML) models were compared. A hybrid model that integrates ML and sample-based statistical techniques for missing value imputation is being proposed. Furthermore, the continuation involved the dataset with the best missing value imputation, chosen based on ML model performance for subsequent feature engineering and hyperparameter tuning. K-means clustering and principal component analysis were applied in our study. Accuracy, the evaluated outcome, improved dramatically and proved that the XGBoost model gives very high accuracy at around 0.125 root mean squared logarithmic error (RMSLE). To overcome overfitting, K-fold cross-validation was implemented.

An Integrated Novel Framework for Coping Missing Values Imputation and Classification

This work presents an integrated framework for imputation of missing values and prediction of class label of unseen samples by using the best features of rule based inductive decision tree (DT) and Support Vector Machine (SVM) classifier (DT-SVM). In this work, the decision tree is used for imputation of missing values of the datasets containing both categorical and numerical valued attributes. In addition, some of the other popular and simple missing value imputation techniques like drop, mean, median, mode, and k-nearest neighbor (kNN) are used for a comparative analysis. The imputed datasets are then classified using SVM. The performance of the proposed integrated novel framework DT-SVM has been compared with Drop-SVM, Mean-SVM, Median-SVM, Mode-SVM, and kNN-SVM and it is found that DT-SVM outperforms others. Further, a new variant of kNN named it as approximated kNN (A-kNN) has been proposed to overcome some of the shortcomings of canonical kNN while learning from a training set imputed by DT. Unlike canonical kNN, A-kNN does not scan the entire training set. Instead, it processes some of the representative instances from the training dataset to identify the nearest neighbor. The class centroid approach is adopted to find the representative instances of the training set. The effectiveness in term of accuracy as well as computational time of A-kNN is examined by comparing with canonical kNN. It is found that computational time of the proposed A-kNN is drastically reduced as compared to canonical kNN without compromising with the classification accuracy.

Kernel weighted least square approach for imputing missing values of metabolomics data

Mass spectrometry is a modern and sophisticated high-throughput analytical technique that enables large-scale metabolomic analyses. It yields a high-dimensional large-scale matrix (samples × metabolites) of quantified data that often contain missing cells in the data matrix as well as outliers that originate for several reasons, including technical and biological sources. Although several missing data imputation techniques are described in the literature, all conventional existing techniques only solve the missing value problems. They do not relieve the problems of outliers. Therefore, outliers in the dataset decrease the accuracy of the imputation. We developed a new kernel weight function-based proposed missing data imputation technique that resolves the problems of missing values and outliers. We evaluated the performance of the proposed method and other conventional and recently developed missing imputation techniques using both artificially generated data and experimentally measured data analysis in both the absence and presence of different rates of outliers. Performances based on both artificial data and real metabolomics data indicate the superiority of our proposed kernel weight-based missing data imputation technique to the existing alternatives. For user convenience, an R package of the proposed kernel weight-based missing value imputation technique was developed, which is available at https://github.com/NishithPaul/tWLSA.

Knowledge discovery from noisy imbalanced and incomplete binary class data

Class imbalance creates a considerable impact on the classification of instances using traditional classifiers. Class imbalance, along with other difficulties, creates a significant impact on recognizing instances of minority class. Researchers work in various directions to mitigate class imbalance effect along with noise as well as missing values in datasets. However, combined studies of noisy class imbalance along with incomplete datasets have not been performed yet. This article contains a detailed analysis of 84 different machine learning models to deal with noisy binary class imbalanced and incomplete data using AUC, G-Mean, and F1-score as performance metrics. This article contains a detailed experiment considering missing value imputation and oversampling techniques. The article contains three comparisons: first missing value imputation techniques in incomplete and binary class imbalanced data, second, resampling techniques in noisy binary class imbalanced data, and third, combined techniques in noisy binary class imbalanced and incomplete data. We conclude that MICE and KNN techniques perform well with an increase in the imbalanced dataset's missing value from the first comparison. In second comparison, the SMOTE-ENN technique performs better than state-of-art in noisy binary class imbalanced datasets, and in the third comparison, we conclude that MICE with SMOTE-ENN technique perform well compared to the rest of the techniques.

A systematic review of machine learning-based missing value imputation techniques

PurposeThe primary aim of this study is to review the studies from different dimensions including type of methods, experimentation setup and evaluation metrics used in the novel approaches proposed for data imputation, particularly in the machine learning (ML) area. This ultimately provides an understanding about how well the proposed framework is evaluated and what type and ratio of missingness are addressed in the proposals. The review questions in this study are (1) what are the ML-based imputation methods studied and proposed during 2010–2020? (2) How the experimentation setup, characteristics of data sets and missingness are employed in these studies? (3) What metrics were used for the evaluation of imputation method?Design/methodology/approachThe review process went through the standard identification, screening and selection process. The initial search on electronic databases for missing value imputation (MVI) based on ML algorithms returned a large number of papers totaling at 2,883. Most of the papers at this stage were not exactly an MVI technique relevant to this study. The literature reviews are first scanned in the title for relevancy, and 306 literature reviews were identified as appropriate. Upon reviewing the abstract text, 151 literature reviews that are not eligible for this study are dropped. This resulted in 155 research papers suitable for full-text review. From this, 117 papers are used in assessment of the review questions.FindingsThis study shows that clustering- and instance-based algorithms are the most proposed MVI methods. Percentage of correct prediction (PCP) and root mean square error (RMSE) are most used evaluation metrics in these studies. For experimentation, majority of the studies sourced the data sets from publicly available data set repositories. A common approach is that the complete data set is set as baseline to evaluate the effectiveness of imputation on the test data sets with artificially induced missingness. The data set size and missingness ratio varied across the experimentations, while missing datatype and mechanism are pertaining to the capability of imputation. Computational expense is a concern, and experimentation using large data sets appears to be a challenge.Originality/valueIt is understood from the review that there is no single universal solution to missing data problem. Variants of ML approaches work well with the missingness based on the characteristics of the data set. Most of the methods reviewed lack generalization with regard to applicability. Another concern related to applicability is the complexity of the formulation and implementation of the algorithm. Imputations based on k-nearest neighbors (kNN) and clustering algorithms which are simple and easy to implement make it popular across various domains.

Clustering based imputation algorithm using unsupervised neural network for enhancing the quality of healthcare data

Historical and real-time healthcare data sets are valuable sources of information for predictive data analytics. However, most of the historical healthcare data sets are overloaded with challenges. One of the most frequently faced challenge is the problem of missing values, occurring because of the inaccuracies in data transmission or data entry processes. An appropriate technique for handling missing values is required to generate good quality data sets for achieving better prediction results. Removing the records with missing values, known as marginalization, poses an easy way out to this challenge. But, this will lessen the data volume of the historical data set and disturb the class balance of the data set. An alternative to marginalization is replacing missing values with plausible values, known as imputation. This paper proposes a missing value imputation technique, CLUSTIMP, using an unsupervised neural network Adaptive Resonance Theory 2 (ART2). The efficiency of the proposed imputation method is evaluated on the incomplete Mammographic mass data set and Hepatocellular Carcinoma data set (HCC) from the UCI repository considering Root Mean Squared Error (RMSE) rate and classification accuracy as the evaluation metrics. The proposed CLUSTIMP imputation algorithm outperforms existing state-of-the-art imputation methods by reducing classifiers error rates between 2 and 11%.

A Bayesian vector autoregression-based data analytics approach to enable irregularly-spaced mixed-frequency traffic collision data imputation with missing values

Traffic collision data are always collected in irregularly spaced and mixed frequency. Conventional treatment on these kinds of data, for instance, aggregating the high-frequency data into the lower frequency, can lead to the loss of relevant information of high-frequency data, and introduce potential temporal instabilities. A novel Bayesian vector autoregression approach is proposed to address this problem. An unevenly-spaced traffic collision data with missing values, containing all collisions in different severities that occurred on the state highways in Washington State from January 2006 to December 2016, is selected in this study the impacts of transportation-, weather- and socioeconomic-related characteristics on traffic collisions. A Gibbs sampler is used to conduct Bayesian inference for model parameters and unobserved high-frequency variables. Results show that the model has a fairly superior fit accuracy, and is able to capture the unobserved heterogeneity in the dataset. The proposed VAR also demonstrates better performance than other missing value imputation techniques, including linear regression, predictive mean matching, k-nearest neighbors, and random forests. This study provides potential in the guidance of model construction that considers the mixed-time-series nature of data.

Missing value imputation: a review and analysis of the literature (2006–2017)

Missing value imputation (MVI) has been studied for several decades being the basic solution method for incomplete dataset problems, specifically those where some data samples contain one or more missing attribute values. This paper aims at reviewing and analyzing related studies carried out in recent decades, from the experimental design perspective. Altogether, 111 journal papers published from 2006 to 2017 are reviewed and analyzed. In addition, several technical issues encountered during the MVI process are addressed, such as the choice of datasets, missing rates and missingness mechanisms, and the MVI techniques and evaluation metrics employed, are discussed. The results of analysis of these issues allow limitations in the existing body of literature to be identified based upon which some directions for future research can be gleaned.

A New Approach of Outlier-robust Missing Value Imputation for Metabolomics Data Analysis

Background: Metabolomics data generation and quantification are different from other types of molecular “omics” data in bioinformatics. Mass spectrometry (MS) based (gas chromatography mass spectrometry (GC-MS), liquid chromatography mass spectrometry (LC-MS), etc.) metabolomics data frequently contain missing values that make some quantitative analysis complex. Typically metabolomics datasets contain 10% to 20% missing values that originate from several reasons, like analytical, computational as well as biological hazard. Imputation of missing values is a very important and interesting issue for further metabolomics data analysis. </P><P> Objective: This paper introduces a new algorithm for missing value imputation in the presence of outliers for metabolomics data analysis. </P><P> Method: Currently, the most well known missing value imputation techniques in metabolomics data are knearest neighbours (kNN), random forest (RF) and zero imputation. However, these techniques are sensitive to outliers. In this paper, we have proposed an outlier robust missing imputation technique by minimizing twoway empirical mean absolute error (MAE) loss function for imputing missing values in metabolomics data. Results: We have investigated the performance of the proposed missing value imputation technique in a comparison of the other traditional imputation techniques using both simulated and real data analysis in the absence and presence of outliers. Conclusion: Results of both simulated and real data analyses show that the proposed outlier robust missing imputation technique is better performer than the traditional missing imputation methods in both absence and presence of outliers.

Missing value imputation for gene expression data by tailored nearest neighbors.

High dimensional data like gene expression and RNA-sequences often contain missing values. The subsequent analysis and results based on these incomplete data can suffer strongly from the presence of these missing values. Several approaches to imputation of missing values in gene expression data have been developed but the task is difficult due to the high dimensionality (number of genes) of the data. Here an imputation procedure is proposed that uses weighted nearest neighbors. Instead of using nearest neighbors defined by a distance that includes all genes the distance is computed for genes that are apt to contribute to the accuracy of imputed values. The method aims at avoiding the curse of dimensionality, which typically occurs if local methods as nearest neighbors are applied in high dimensional settings. The proposed weighted nearest neighbors algorithm is compared to existing missing value imputation techniques like mean imputation, KNNimpute and the recently proposed imputation by random forests. We use RNA-sequence and microarray data from studies on human cancer to compare the performance of the methods. The results from simulations as well as real studies show that the weighted distance procedure can successfully handle missing values for high dimensional data structures where the number of predictors is larger than the number of samples. The method typically outperforms the considered competitors.

Metabolomic Biomarker Identification in Presence of Outliers and Missing Values.

Metabolomics is the sophisticated and high-throughput technology based on the entire set of metabolites which is known as the connector between genotypes and phenotypes. For any phenotypic changes, potential metabolite (biomarker) identification is very important because it provides diagnostic as well as prognostic markers and can help to develop new biomolecular therapy. Biomarker identification from metabolomics data analysis is hampered by the use of high-throughput technology that provides high dimensional data matrix which contains missing values as well as outliers. However, missing value imputation and outliers handling techniques play important role in identifying biomarker correctly. Although several missing value imputation techniques are available, outliers deteriorate the accuracy of imputation as well as the accuracy of biomarker identification. Therefore, in this paper we have proposed a new biomarker identification technique combining the groupwise robust singular value decomposition, t-test, and fold-change approach that can identify biomarkers more correctly from metabolomics dataset. We have also compared the performance of the proposed technique with those of other traditional techniques for biomarker identification using both simulated and real data analysis in absence and presence of outliers. Using our proposed method in hepatocellular carcinoma (HCC) dataset, we have also identified the four upregulated and two downregulated metabolites as potential metabolomic biomarkers for HCC disease.

FIMUS: A framework for imputing missing values using co-appearance, correlation and similarity analysis

Natural data sets often have missing values in them. An accurate missing value imputation is crucial to increase the usability of a data set for statistical analyses and data mining tasks. In this paper we present a novel missing value imputation technique using a data set’s existing patterns including co-appearances of attribute values, correlations among the attributes and similarity of values belonging to an attribute. Our technique can impute both numerical and categorical missing values. We carry out extensive experiments on nine natural data sets, and compare our technique with four high quality existing techniques. We simulate 32 types of missing patterns (combinations), and thereby generate 320 missing data sets for each of the nine natural data sets. Two well known evaluation criteria namely index of agreement (d2) and root mean squared error are used. Our experimental results, based on the statistical sign test, indicate that our technique achieves significantly better imputation accuracy than the existing techniques.

A Novel Approach for Missing Value Imputation and Classification of Microarray Dataset

Various machine learning algorithms are used for classification of microarray data. The accuracy of classifier is generally compromised by the missing value in the data set, due to incorrect collection of data values. Those missing data in the microarray data set may contain some relevance information. These missing values can lead to the incorrect analysis of data. Lots of research is carried out to minimize the impact of these missing values. In this paper; a new method has been proposed to elucidate the missing value for different microarray data sets. The proposed method has been compared with the traditional methods like mean method and median method. The proposed model has used principal component analysis (PCA) to deal with curse of dimensionality. To validate the proposed method Linear Discriminant Analysis (LDA) classification technique has been used to achieve the accuracy. The results show that proposed missing value imputation technique gives more accuracy as compared to other missing value methods.