Percentage Of Missing Values Research Articles

Bayesian Nonparametric Classification for Incomplete Data With a High Missing Rate: an Application to Semiconductor Manufacturing Data

During the semiconductor manufacturing process, predicting the yield of the semiconductor is an important problem. Early detection of defective product production in the manufacturing process can save huge production cost. The data generated from the semiconductor manufacturing process have characteristics of non-normal distributions, random missing patterns and high missing rate, which complicate the prediction of the yield. We propose the Dirichlet Process -Naive Bayes model (DPNB) that can simultaneously impute missing values and address classification problems. Since the DPNB is based on the infinite Gaussian mixture model, it can estimate complex data distributions and make predictions for missing datasets with some missing patterns due to nice properties of the Gaussian distribution. The DPNB also performs well for high missing rates since it uses all information of observed components. Experiments on various real datasets including semiconductor manufacturing data show that the DPNB has better performance than state-of-the-art methods in terms of predicting missing values and target variables as percentage of missing values increases.

Towards Improving Predictive Statistical Learning Model Accuracy by Enhancing Learning Technique

The accuracy of the statistical learning model depends on the learning technique used which in turn depends on the dataset’s values. In most research studies, the existence of missing values (MVs) is a vital problem. In addition, any dataset with MVs cannot be used for further analysis or with any data driven tool especially when the percentage of MVs are high. In this paper, the authors propose a novel algorithm for dealing with MVs depending on the feature selection (FS) of similarity classifier with fuzzy entropy measure. The proposed algorithm imputes MVs in cumulative order. The candidate feature to be manipulated is selected using similarity classifier with Parkash’s fuzzy entropy measure. The predictive model to predict MVs within the candidate feature is the Bayesian Ridge Regression (BRR) technique. Furthermore, any imputed features will be incorporated within the BRR equation to impute the MVs in the next chosen incomplete feature. The proposed algorithm was compared against some practical state-of-the-art imputation methods by conducting an experiment on four medical datasets which were gathered from several databases repository with MVs generated from the three missingness mechanisms. The evaluation metrics of mean absolute error (MAE), root mean square error (RMSE) and coefficient of determination (R2 score) were used to measure the performance. The results exhibited that performance vary depending on the size of the dataset, amount of MVs and the missingness mechanism type. Moreover, compared to other methods, the results showed that the proposed method gives better accuracy and less error in most cases.

The effect of simple imputations based on four variants of PCA methods on the quantiles of annual rainfall data.

Hydrology-related studies often require complete datasets. However, missing data is an unavoidable reality. In this regard, the imputed data could fulfill the same role as the observed ones, while they are uncertain and just estimated. The aim of this study is to compare the performance of four simple imputation variants derived from the principal component analysis (PCA) for imputing annual total rainfall series obtained from stations located in northeast Algeria. On the other hand, the study focuses on the effects on quantiles of annual rainfall data due to imputations by the former methods. The four variants are probabilistic PCA, expectation maximization PCA, regularized PCA, and singular value decomposition PCA. Annual rainfall data from 30 stations for the period ranging from 1935 to 2004 (69years) are used to generate and impute gaps for four different percentages of missing values (PMV), namely, 10, 20, 30, and 40%. Based on some well-known statistical indices, the results show that the regularized PCA and expectation maximization PCA variants perform better than the other imputation methods considered in this study and result in very good to acceptable predicted quantiles, such as the following: correlation coefficient is equal to 0.97 with 10% of percentage of missing values and 0.66 with 40%; the relative error between observed and predicted quantiles is equal to 4.74% with 10% of percentage of missing values and 3.82% with 40%.

In-depth method assessments of differentially expressed protein detection for shotgun proteomics data with missing values

Considering as one of the major goals in quantitative proteomics, detection of the differentially expressed proteins (DEPs) plays an important role in biomarker selection and clinical diagnostics. There have been plenty of algorithms and tools focusing on DEP detection in proteomics research. However, due to the different application scopes of these methods, and various kinds of experiment designs, it is not very apparent about the best choice for large-scale proteomics data analyses. Moreover, given the fact that proteomics data usually contain high percentage of missing values (MVs), but few replicates, a systematic evaluation of the DEP detection methods combined with the MV imputation methods is essential and urgent. Here, we analyzed a total of four representative imputation methods and five DEP methods on different experimental and simulated datasets. The results showed that (i) MV imputation could not always improve the performances of DEP detection methods and the imputation effects differed in the missing value percentages; (ii) the DEP detection methods had different statistical powers affected by the percentage of MVs. Two statistical methods (i.e. the empirical Bayesian random censoring threshold model, and the significance analysis of microarray) performed better than the other evaluated methods in terms of accuracy and sensitivity.

Handling varying amounts of missing data when classifying mental-health risk levels.

One of the main challenges of classifying clinical data is determining how to handle missing features. Most research favours imputing of missing values or neglecting records that include missing data, both of which can degrade accuracy when missing values exceed a certain level. In this research we propose a methodology to handle data sets with a large percentage of missing values and with high variability in which particular data are missing. Feature selection is effected by picking variables sequentially in order of maximum correlation with the dependent variable and minimum correlation with variables already selected. Classification models are generated individually for each test case based on its particular feature set and the matching data values available in the training population. The method was applied to real patients' anonymous mental-health data where the task was to predict the suicide risk judgement clinicians would give for each patient's data, with eleven possible outcome classes: zero to ten, representing no risk to maximum risk. The results compare favourably with alternative methods and have the advantage of ensuring explanations of risk are based only on the data given, not imputed data. This is important for clinical decision support systems using human expertise for modelling and explaining predictions.

Estimating the reliability coefficient of tests in presence of missing values.

The problem of missing values at the item level is common in studies using educational and psychological tests. The aim of the present work is to explore how the estimation of reliability is affected by missing values. Using real data, we simulated missing values in accordance with a "missing at random mechanism". Four factors were manipulated with the aim of checking their effect on the estimation of the reliability of the instrument: missing data mechanism, percentage of missing data in the database, sample size, and procedure employed for the treatment of missing values. The results show that the quality of estimations depends on the interaction of various factors. The general tendency is that the estimations are worse when the sample size is small and the percentage of missing values increases. Listwise is the worst procedure for treatment of the missing data in the simulated conditions. It is concluded that with a small percentage of missing values one can obtain estimations that are acceptable from a practical point of view with all the procedures employed, except Listwise.

Analysis of longitudinal binary outcomes in clinical trials with low percentage of missing values

In interventional or observational longitudinal studies, the issue of missing values is one of the main concepts that should be investigated. The researcher's main concerns are the impact of missing data on the final results of the study and the appropriate methods that missing values should be handled. Regarding the role and the scale of the variable that missing values have been occurred and the structure of missing values, different methods for analysis have been presented. In this article, the impact of missing values on a binary response variable, in a longitudinal clinical trial with three follow up sessions has been investigated Propensity Score, Predictive Model Based and Mahalanobis imputation strategies with complete case and available data methods have been used for dealing with missing values in the mentioned study. Three models; Random intercept, Marginal GEE and Marginalized Random effects models were implemented to evaluate the effect of covariates. The percentage of missing responses in each of the treatment groups, throughout the course of the study, differs from 6.8 to 14.1. Although, the estimate of variance component in random intercept and marginalized random effect models were highly significant (p <0.001) the same results were obtained for the effect of independent variables on the response variable with different imputation strategies. In our study according to the low missing percentage, there were no considerable differences between different methods that were used for handling missing data.