Missing Value Imputation Approach Research Articles

Predicting insurance claims through a variety of data mining techniques: facing lots of missing values and moderate class-imbalanced levels

Abstract This paper copes with a real-world classification problem related to the management of claims received in an insurance company. The way to obtain the classifier is not easy due to the high amount of missing values as well as the inherent imbalanced scenario within class labels. Once the data partition has been done, the training set is submitted to an intensive double grid search in order to obtain the most promising type of missing value imputation approach and then a step ahead is done using the best method and it starts the next round of data mining strategies which now falls into data rebalancing umbrella. Again, a grid search from an undersampling and oversampling family with different settings is done taking into account only seen data. The training data obtained after the first grid search are now submitted to the second step according the second grid search in order to get the ready training set for the further classifier training. The main objective of the work is to find the best combination of data mining techniques that suits the data set with a pipeline containing two types of data preparation methods coming from different families. As an outcome, first the problem of the presence of missing values has been addressed and then the data rebalancing techniques has been applied. The study focuses on obtaining classifiers based on Bayesian and lazy approaches as well as decision trees, evaluated on metrics such as the area under the ROC curve (AUC), Cohen’s kappa, Accuracy and the F-measure, among others. The imputation by the mean the mode is preferable to the Expectation Maximization Imputation in the scenario faced in this paper taking into account that the amount of missing values is higher than a forty percent for many features.

Remora based Deep Maxout Network model for network intrusion detection using Convolutional Neural Network features

With the rapid evolution of the Internet of Things (IoT), network advancement has significantly influenced the increasing number of devices and advanced enhancements linked with it. Indeed the increasing number prevalence and sophistication of emerging cyber-attacks have highlighted the necessity for designing robust security application. In this paper, the Remora-based Deep Maxout Network model is Proposed. Here, the input data is acquired and forwarded to the pre-processing phase, wherein the missing value imputation approach is employed for creating a complete dataset. Later, the pre-processed data is then subjected to dimension transformation from the transformed data; the Convolutional Neural Network features are extracted, followed by feature selection based on Canberra distance. Here, detection is carried out using a Deep Maxout Network whose weights and training parameters are modified using the Remora Optimization Algorithm. However, the proposed model has delivered superior results with a high testing accuracy of 0.945

An Improved Bi-LSTM-Based Missing Value Imputation Approach for Pregnancy Examination Data

In recent years, the development of computer technology has promoted the informatization and intelligentization of hospital management systems and thus produced a large amount of medical data. These medical data are valuable resources for research. We can obtain inducers and unknown symptoms that can help discover diseases and make earlier diagnoses. Hypertensive disorder in pregnancy (HDP) is a common obstetric complication in pregnant women, which has severe adverse effects on the life safety of pregnant women and fetuses. However, the early and mid-term symptoms of HDP are not obvious, and there is no effective solution for it except for terminating the pregnancy. Therefore, detecting and preventing HDP is of great importance. This study aims at the preprocessing of pregnancy examination data, which serves as a part of HDP prediction. We found that the problem of missing data has a large impact on HDP prediction. Unlike general data, pregnancy examination data have high dimension and a high missing rate, are in a time series, and often have many non-linear relations. Current methods are not able to process the data effectively. To this end, we propose an improved bi-LSTM-based missing value imputation approach. It combines traditional machine learning and bidirectional LSTM to deal with missing data of pregnancy examination data. Our missing value imputation method obtains a good effect and improves the accuracy of the later prediction of HDP using examination data.

Optimal method for determining the intraclass correlation coefficients of urinary biomarkers such as dialkylphosphates from imputed data

Urinary biomarkers are commonly used in epidemiological studies as surrogates or indicators of exposure to chemical substances. Evaluating the reliability of a biomarker is highly important because use of an unreliable marker may lead to misclassification and attenuation bias, resulting in flawed interpretations and conclusions. Although intraclass correlation coefficient (ICC) is regarded as a typical index of test reliability, methods for determining the ICCs of urinary biomarkers have not been standardised, and different methods have been used. This study evaluated different imputation methods for left-censored data, i.e., four imputation or one substitution methods, before calculating ICCs, and at the same time mathematically assessed the impact of the left-censoring proportion on the estimated ICCs. Biomarkers of exposure to organophosphate pesticides, i.e., dialkylphosphates, were used as an example. The Gibbs sampler-based left-censored missing value imputation approach had the best performance for imputation of values below reporting limits, with lower values on Kolmogorov–Smirnov test statistics than other imputation/substitution methods, i.e., a univariate distribution fitting approach, multiple imputation by chained equation, a bootstrap expectation–maximisation algorithm approach, and a single value substitution. In all imputation methods, however, ICCs decreased as censoring rates increased. We propose a method to estimate true ICCs based on mathematical estimation.

Improved GSimp: A Flexible Missing Value Imputation Method to Support Regulatory Bioequivalence Assessment.

Missing values are not uncommon in in vivo bioequivalence (BE) studies and pose non-trivial challenges for BE assessment. Missing values typically appear as a mixture of different types, such as Missing Not at Random (MNAR) and Missing Completely at Random (MCAR), however, current data imputation methods were usually developed for a certain type of missing values (e.g., MNAR). Among them, an iterative Gibbs sampler-based left-censored missing value imputation approach (GSimp) was recently developed and showed superior performance over other methods in handling MNAR data. In this study, we introduce an improved GSimp ("Improved GSimp" thereafter) that offers flexibility in handling mixed types of missing data and better imputation accuracy to support BE assessment for studies with missing values. Simulations mimicking different missing value scenarios (e.g., mixture of different missing types and proportion of missing values) were conducted to compare performance of the Improved GSimp with other methods (e.g., original GSimp and half of minimal value). Normalized root mean square error (NRMSE) was used to evaluate imputation accuracy. Our results showed that the Improved GSimp always had the best accuracy in all simulated scenarios compared to other methods.

A Pragmatic Ensemble Strategy for Missing Values Imputation in Health Records.

Pristine and trustworthy data are required for efficient computer modelling for medical decision-making, yet data in medical care is frequently missing. As a result, missing values may occur not just in training data but also in testing data that might contain a single undiagnosed episode or a participant. This study evaluates different imputation and regression procedures identified based on regressor performance and computational expense to fix the issues of missing values in both training and testing datasets. In the context of healthcare, several procedures are introduced for dealing with missing values. However, there is still a discussion concerning which imputation strategies are better in specific cases. This research proposes an ensemble imputation model that is educated to use a combination of simple mean imputation, k-nearest neighbour imputation, and iterative imputation methods, and then leverages them in a manner where the ideal imputation strategy is opted among them based on attribute correlations on missing value features. We introduce a unique Ensemble Strategy for Missing Value to analyse healthcare data with considerable missing values to identify unbiased and accurate prediction statistical modelling. The performance metrics have been generated using the eXtreme gradient boosting regressor, random forest regressor, and support vector regressor. The current study uses real-world healthcare data to conduct experiments and simulations of data with varying feature-wise missing frequencies indicating that the proposed technique surpasses standard missing value imputation approaches as well as the approach of dropping records holding missing values in terms of accuracy.

Multiple Imputation Approaches Applied to the Missing Value Problem in Bottom-Up Proteomics.

Analysis of differential abundance in proteomics data sets requires careful application of missing value imputation. Missing abundance values widely vary when performing comparisons across different sample treatments. For example, one would expect a consistent rate of “missing at random” (MAR) across batches of samples and varying rates of “missing not at random” (MNAR) depending on the inherent difference in sample treatments within the study. The missing value imputation strategy must thus be selected that best accounts for both MAR and MNAR simultaneously. Several important issues must be considered when deciding the appropriate missing value imputation strategy: (1) when it is appropriate to impute data; (2) how to choose a method that reflects the combinatorial manner of MAR and MNAR that occurs in an experiment. This paper provides an evaluation of missing value imputation strategies used in proteomics and presents a case for the use of hybrid left-censored missing value imputation approaches that can handle the MNAR problem common to proteomics data.

Left-Censored Missing Value Imputation Approach for MS-Based Proteomics Data with GSimp.

Missing values caused by the limit of detection or quantification (LOD/LOQ) were widely observed in mass spectrometry (MS)-based omics studies and could be recognized as missing not at random (MNAR). MNAR leads to biased statistical estimations and jeopardizes downstream analyses. Although a wide range of missing value imputation methods was developed for omics studies, a limited number of methods were designed appropriately for the situation of MNAR. To facilitate MS-based omics studies, we introduce GSimp, a Gibbs sampler-based missing value imputation approach, to deal with left-censor missing values in MS-proteomics datasets. In this book, we explain the MNAR and elucidate the usage of GSimp for MNAR in detail.

An investigation of solutions for handling incomplete online review datasets with missing values

ABSTRACT Online review helpfulness prediction is an important research issue in electronic commerce and data mining. However, the collected datasets used for the analysis and prediction of the helpfulness of online reviews often contain some missing attribute values, such as reviewer background and rating information. In related literatures, many studies have either used the case deletion approach to remove the data containing missing values or considered the imputation of missing values by the mean/mode method. However, none of them consider the direct handling approach without missing value imputation for online review datasets by decision tree-related techniques. Therefore, in this paper, we investigate the suitability of different types of approaches to solve the incomplete dataset problem of online reviews. Specifically, for missing value imputation, several supervised learning techniques including MICE, KNN, SVM, and CART are examined. Moreover, for the direct handling approach without missing value imputation, CART is also performed for this task. The experimental results based on the TripAdvisor dataset for review helpfulness prediction show that the approach where incomplete online review datasets are handled directly without imputation by CART significantly outperforms the other approaches, including case deletion and missing value imputation approaches.

Missing value imputation through shorter interval selection driven by Fuzzy C-Means clustering

The presence of missing data is a common and pivotal issue, which generally leads to a serious decrease of data quality and thus indicates the necessity to effectively handle missing data. In this paper, we propose a missing value imputation approach driven by Fuzzy C-Mean clustering to improve the classification accuracy by referring only to the known feature values of some selected instances. In particular, the missing values for each instance are imputed by selecting a shorter interval based on the cluster membership value within the certain threshold limit of each feature, while using a short interval is considered to improve the imputation effectiveness and get more accurate estimation of the values in comparison with using a long interval. Our method is evaluated through comparing with state-of-the-art imputation methods on UCI datasets. The experimental results demonstrate that the proposed approach performs closely to or better than those state-of-the-art imputation methods.

A data-driven missing value imputation approach for longitudinal datasets

Longitudinal datasets of human ageing studies usually have a high volume of missing data, and one way to handle missing values in a dataset is to replace them with estimations. However, there are many methods to estimate missing values, and no single method is the best for all datasets. In this article, we propose a data-driven missing value imputation approach that performs a feature-wise selection of the best imputation method, using known information in the dataset to rank the five methods we selected, based on their estimation error rates. We evaluated the proposed approach in two sets of experiments: a classifier-independent scenario, where we compared the applicabilities and error rates of each imputation method; and a classifier-dependent scenario, where we compared the predictive accuracy of Random Forest classifiers generated with datasets prepared using each imputation method and a baseline approach of doing no imputation (letting the classification algorithm handle the missing values internally). Based on our results from both sets of experiments, we concluded that the proposed data-driven missing value imputation approach generally resulted in models with more accurate estimations for missing data and better performing classifiers, in longitudinal datasets of human ageing. We also observed that imputation methods devised specifically for longitudinal data had very accurate estimations. This reinforces the idea that using the temporal information intrinsic to longitudinal data is a worthwhile endeavour for machine learning applications, and that can be achieved through the proposed data-driven approach.

An Exploration of Online Missing Value Imputation in Non-stationary Data Stream

Missing value imputation (MVI) is an important data preprocessing technique. In previous decades, MVI technique has been widely studied as well as most MVI approaches have been proposed by means of either statistics or machine learning techniques. However, all previous methods only focus on the static data, but ignore the imputation for the dynamic online data. It is intuitionistic that the imputation errors may be significantly increased when there exists concept drifts in the data stream. In this paper, we investigate the impact of adopting the conventional MVI methods in non-stationary data stream. Meanwhile, two slide time window-based strategies are proposed to alleviate this impact, where one is the plain average strategy, and the other is the logarithmic weighted average strategy that gradually adds the weights of instances along the time axis. Combining with the proposed strategies, three popular MVI techniques, mean imputation (MI), KNN imputation (KNNI) and the Bayesian principal component analysis imputation (BPCAI) are adopted, to indicate the effect of the strategies are irrelevant to the specific MVI technique. The experimental results on three different types' concept drift synthetic data sets and two real-world drifting data sets have presented the effectiveness and feasibility of the proposed strategies. Moreover, the impact of time window size has also been investigated for guiding the parameter settings in future practical applications.

CBRG: A Novel Algorithm for Handling Missing Data Using Bayesian Ridge Regression and Feature Selection Based on Gain Ratio

Existing imputation methods may lead to biased predictions and decrease or increase the statistical influence which leads to improper estimations. Several missing value imputation approaches performance depends on the size of the dataset and the number of missing values within the dataset. In this work, the authors proposed a novel algorithm for manipulating missing data versus some common imputation approaches. The proposed algorithm imputes missing values in cumulative order depending on the gain ratio (GR) feature selection (to select the candidate feature to be manipulated) and the Bayesian Ridge Regression (BRR) technique (to build the predictive model). Each imputed feature will be used to manipulate the missing values in the following selected candidate feature. The proposed algorithm was implemented on eight different datasets after generating different missing values proportions from the missingness mechanisms. The imputation performance was calculated in terms of imputation time, mean absolute error (MAE), coefficient of determination ( $R^{2}$ ), and root-mean-square error (RMSE). The results show the efficiency of the proposed algorithm when imputing any dataset with any number of missing data from any missingness mechanism.

Enhanced Application of Principal Component Analysis in Machine Learning for Imputation of Missing Traffic Data

Missing value imputation approaches have been widely used to support and maintain the quality of traffic data. Although the spatiotemporal dependency-based approaches can improve the imputation performance for large and continuous missing patterns, additionally considering traffic states can lead to more reliable results. In order to improve the imputation performances further, a section-based approach is also needed. This study proposes a novel approach for identifying traffic-states of different spots of road sections that comprise, namely, a section-based traffic state (SBTS), and determining their spatiotemporal dependencies customized for each SBTS, for missing value imputations. A principal component analysis (PCA) was employed, and angles obtained from the first principal component were used to identify the SBTSs. The pre-processing was combined with a support vector machine for developing the imputation model. It was found that the segmentation of the SBTS using the angles and considering the spatiotemporal dependency for each state by the proposed approach outperformed other existing models.

GSimp: A Gibbs sampler based left-censored missing value imputation approach for metabolomics studies.

Left-censored missing values commonly exist in targeted metabolomics datasets and can be considered as missing not at random (MNAR). Improper data processing procedures for missing values will cause adverse impacts on subsequent statistical analyses. However, few imputation methods have been developed and applied to the situation of MNAR in the field of metabolomics. Thus, a practical left-censored missing value imputation method is urgently needed. We developed an iterative Gibbs sampler based left-censored missing value imputation approach (GSimp). We compared GSimp with other three imputation methods on two real-world targeted metabolomics datasets and one simulation dataset using our imputation evaluation pipeline. The results show that GSimp outperforms other imputation methods in terms of imputation accuracy, observation distribution, univariate and multivariate analyses, and statistical sensitivity. Additionally, a parallel version of GSimp was developed for dealing with large scale metabolomics datasets. The R code for GSimp, evaluation pipeline, tutorial, real-world and simulated targeted metabolomics datasets are available at: https://github.com/WandeRum/GSimp.

Missing Value Imputation Approach for Mass Spectrometry-based Metabolomics Data

Missing values exist widely in mass-spectrometry (MS) based metabolomics data. Various methods have been applied for handling missing values, but the selection can significantly affect following data analyses. Typically, there are three types of missing values, missing not at random (MNAR), missing at random (MAR), and missing completely at random (MCAR). Our study comprehensively compared eight imputation methods (zero, half minimum (HM), mean, median, random forest (RF), singular value decomposition (SVD), k-nearest neighbors (kNN), and quantile regression imputation of left-censored data (QRILC)) for different types of missing values using four metabolomics datasets. Normalized root mean squared error (NRMSE) and NRMSE-based sum of ranks (SOR) were applied to evaluate imputation accuracy. Principal component analysis (PCA)/partial least squares (PLS)-Procrustes analysis were used to evaluate the overall sample distribution. Student’s t-test followed by correlation analysis was conducted to evaluate the effects on univariate statistics. Our findings demonstrated that RF performed the best for MCAR/MAR and QRILC was the favored one for left-censored MNAR. Finally, we proposed a comprehensive strategy and developed a public-accessible web-tool for the application of missing value imputation in metabolomics (https://metabolomics.cc.hawaii.edu/software/MetImp/).

Scalable Data Quality for Big Data: The Pythia Framework for Handling Missing Values

Solving the missing-value (MV) problem with small estimation errors in large-scale data environments is a notoriously resource-demanding task. The most widely used MV imputation approaches are computationally expensive because they explicitly depend on the volume and the dimension of the data. Moreover, as datasets and their user community continuously grow, the problem can only be exacerbated. In an attempt to deal with such a problem, in our previous work, we introduced a novel framework coined Pythia, which employs a number of distributed data nodes (cohorts), each of which contains a partition of the original dataset. To perform MV imputation, the Pythia, based on specific machine and statistical learning structures (signatures), selects the most appropriate subset of cohorts to perform locally a missing value substitution algorithm (MVA). This selection relies on the principle that particular subset of cohorts maintains the most relevant partition of the dataset. In addition to this, as Pythia uses only part of the dataset for imputation and accesses different cohorts in parallel, it improves efficiency, scalability, and accuracy compared to a single machine (coined Godzilla), which uses the entire massive dataset to compute imputation requests. Although this article is an extension of our previous work, we particularly investigate the robustness of the Pythia framework and show that the Pythia is independent from any MVA and signature construction algorithms. In order to facilitate our research, we considered two well-known MVAs (namely K-nearest neighbor and expectation-maximization imputation algorithms), as well as two machine and neural computational learning signature construction algorithms based on adaptive vector quantization and competitive learning. We prove comprehensive experiments to assess the performance of the Pythia against Godzilla and showcase the benefits stemmed from this framework.

APPLICATION OF AN EMPIRICAL GROWTH MODEL AND MULTIPLE IMPUTATION IN HARD DISK DRIVE FIELD RETURN PREDICTION

Appropriate model assumption and robust model estimation are critical for accurate hard disk drive (HDD) field return prediction. Parametric distribution models that are conventionally used in HDD reliability analyses do not describe well HDD field survival. A biological growth model, Bertalanfy-Richards growth model, is introduced in this paper to model HDD field survival within warranty period. A nonparametric multiple missing value imputation approach is used to improve model estimation when not enough observations of field returns are available at early stage of product life. Tests of the model and the missing value imputation approach using actual HDD field survival data suggest they perform well in describing and predicting HDD warranty field returns.

Missing value imputation improves clustering and interpretation of gene expression microarray data

BackgroundMissing values frequently pose problems in gene expression microarray experiments as they can hinder downstream analysis of the datasets. While several missing value imputation approaches are available to the microarray users and new ones are constantly being developed, there is no general consensus on how to choose between the different methods since their performance seems to vary drastically depending on the dataset being used.ResultsWe show that this discrepancy can mostly be attributed to the way in which imputation methods have traditionally been developed and evaluated. By comparing a number of advanced imputation methods on recent microarray datasets, we show that even when there are marked differences in the measurement-level imputation accuracies across the datasets, these differences become negligible when the methods are evaluated in terms of how well they can reproduce the original gene clusters or their biological interpretations. Regardless of the evaluation approach, however, imputation always gave better results than ignoring missing data points or replacing them with zeros or average values, emphasizing the continued importance of using more advanced imputation methods.ConclusionThe results demonstrate that, while missing values are still severely complicating microarray data analysis, their impact on the discovery of biologically meaningful gene groups can – up to a certain degree – be reduced by using readily available and relatively fast imputation methods, such as the Bayesian Principal Components Algorithm (BPCA).