Missing Data Problem Research Articles

Estimating the Mean of PM2.5 with Missing Data in the Area Around Electricity Generating Authority of Thailand Using the Improved Compromised Imputation Method

Particulate matter with an aerodynamic diameter of less than 2.5 m or , is one of the air pollutants that has been found to be at unsafe levels for a number of years in Thailand, leading to public health concerns. In order to lessen the detrimental effects of air pollution, monitoring and analysis of concentration are crucial. Following the study of data from the Pollution Control Department Report in the area around the Electricity Generating Authority of Thailand in January 2019, it was found that there was data missing in the information. It is well-known that missing data can reduce the accuracy of data analysis. To solve the missing data problem, this paper proposes an improved method of compromised imputation and a corresponding resultant estimator to deal with estimating the mean of concentrations in the area. The bias and mean square error of the estimator obtained from the proposed method were derived. The conditions which favor the performance of our estimator over other estimators obtained from the mean, ratio, and compromised imputation methods were obtained using mean square error to apply in the area. The mean of concentrations in this case using the proposed estimator was equal to 47.13 g/m3, indicating that it did not exceed unsafe levels (<50 g/m3) under certain conditions. In order to support more accurate data analysis that will lead to effective management of air pollution problems in the future, this research proposes a new method that is more effective than the existing methods under missing data problem.

Modeling geostatistical incomplete spatially correlated survival data with applications to COVID-19 mortality in Ghana

Survival models which incorporate frailties are common in time-to-event data collected over distinct spatial regions. While incomplete data are unavoidable and a common complication in statistical analysis of spatial survival research, most researchers still ignore the missing data problem. In this paper, we propose a geostatistical modeling approach for incomplete spatially correlated survival data. We achieve this by exploring missingness in outcome, covariates, and spatial locations. In the process, we analyze incomplete spatially-referenced survival data using a Weibull model for the baseline hazard function and correlated log-Gaussian frailties to model spatial correlation. We illustrate the proposed method with simulated data and an application to geo-referenced COVID-19 data from Ghana. There are several disagreements between parameter estimates and credible intervals widths obtained using our proposed approach and complete case analysis. Based on these findings, we argue that our approach provides more reliable parameter estimates and has higher predictive accuracy.

Restoration of missing or low-quality 12-lead ECG signals using ensemble deep-learning model with optimal combination

Background and ObjectiveIn a 12-lead electrocardiogram (ECG) examination, the ECG signals often have low-quality data problems due to high-frequency noise caused by muscles and low-frequency noise caused by body movement, breathing. These problems cause delays in examination results and increase medical costs. For this reason, solving low-quality data and missing ECG data problems can provide patients with improved medical services, reducing the work-loss and medical costs. The purpose of this study is to develop a signal restoration model for each of the 12 signals to solve the low-quality and missing data problems caused by mechanical and operator errors during 12-lead ECG examinations. MethodsFor this study, 13,862 high-quality 12-lead ECG recordings for multiple diseases were obtained from the 12-lead ECG database of a general hospital from 2016 to 2020. Two strategies were adopted to develop an accurate restoration model. First, to obtain the optimal input parameters for the ECG regeneration model for each ECG signal, linear regression (LR) models were developed for all 165 three-signal combinations of 11 signals. Second, the restoration models were constructed in a parallel architecture combining bidirectional long short-term memory (Bi-LSTM) with a convolutional neural network (CNN) to learn the temporal and spatial features of optimal combinations. ResultsThe performances of the 165 candidate combinations for restoring missing signal were analyzed through the LR model to find the optimal input parameter for all ECG signals. The average root mean square error of the optimal combinations was 0.082 μV. The average RMSE of the signal restoration model made using the optimal combinations and deep-learning model (Bi-LSTM&CNN) was 0.037 μV, and the cosine simplicity was 0.991. ConclusionsThis ECG restoration technology obtained optimal input parameters through the LR model and developed ECG restoration model through the Bi-LSTM&CNN combined model to restore ECG signals for multiple diseases. The 12-lead ECG signal restoration model developed through this study offers high accuracy for the magnitude and direction components of all 12 signals. This technology can be used in emergency medical systems and remote ECG measurement situations, as well as in synthetic ECG generation technologies for constructing research datasets.

An enhanced dynamic soft sensor–based online estimation of missing data for water distribution system with inherent disturbances

Interruption of water supply is an unfavourable event in water distribution system (WDS). The interruptions are mainly due to water hammer (WH) transients that are caused by sudden pump/valve failures. In this paper, the effects of hammer are mathematically realized considering the effect of valve coefficient integrated with the system. Due to economic and technical limitations, the system is generally equipped with some obsolete instruments for continuous monitoring, which creates a missing data problem. The problem of missing data estimation could be addressed using a novel customized Kalman filter (CKF), a customized version of conventional Kalman filter (KF) based on process model. Initially, to avoid the intricacies of obtaining the optimal estimates of missing instants, the vacancies in the data are filled assigned by the average values of available adjacent data. The measured quantities are considered distorted with state-, input-dependent and independent uncertainties. The effect of WH on a prototypical WDS is studied for different modes of operation, and the missing data points are reconstructed for regular and irregular sampled data set using the proposed CKF. The performance of the proposed algorithm is also compared with the conventional extended Kalman filter (EKF). The study indicates that the proposed missing data estimation algorithm is highly reliable to identify the missing instances in the states of distribution system.

Imputed quantile tensor regression for near-sited spatial-temporal data

Modern spatial temporal data are collected from sensor networks. Missing data problems are common for this kind of data. Making robust and accurate imputation is important in many applications. There are complex correlations in both spatial and temporal dimensions. Thus, it is even a challenge to model missing spatial-temporal data. In this article, the imputation of missing values is with the help of related covariates. First, we transform the original sensor × time observational matrix to a high order tensor by adding an extra temporal dimension. Then we integrate quantile tensor regression with tensor completion. The objective function includes check loss and nuclear norm penalty. An alternating update algorithm combined with alternating direction method of multipliers (ADMM) is developed to solve the objective function. Theoretical properties of the proposed estimator are investigated. Simulation studies show our proposed method is more robust and can get more accurate imputation results. Real data analysis about Beijing's PM2.5 concentration level is conducted to verify the efficiency of the estimation procedure.

Imputed mean tensor regression for near-sited spatial temporal data

Modern spatial temporal data are often collected from sensor networks. Missing data problems are common to this kind of data. Making accurate imputation is important for many applications. In the unsupervised setting, one technique is to minimize the rank of a tensor or matrix. If we add related covariates, can we get more accurate imputation results? To address this, we transform the original sensor×time measurements to high order tensors by adding additional temporal dimensions and then integrate tensor regression with tensor completion using nuclear norm penalty. One advantage is we can simultaneously estimate parameters and impute missing values due to clear spatial consistency for near-sited spatial-temporal data. The proposed method doesn't assume missing mechanism of the response. Theoretical properties of the proposed estimator are investigated. Simulation studies and real data analysis are conducted to verify the efficiency of the estimation procedure.

Deep Learning‐Based Multiomics Data Integration Methods for Biomedical Application

The innovation of high‐throughput technologies and medical radiomics allows biomedical data to accumulate at an astonishing rate. Several promising deep learning (DL) methods are developed to integrate multiomics data generated from a large number of samples. Herein, a comprehensive survey is conducted and the state‐of‐the‐art DL‐based multiomics data integration methods in the biomedical field are reviewed. These methods are classified into six categories according to their model framework, and the specific applicable scenarios of each category are summarized in five biomedicine aspects. DL‐based methods offer opportunities for disentangling biomolecular mechanisms in biomedical applications. There are, however, limitations with these methods, such as missing data problem and “black‐box” nature. A discussion of some of the recommendations for these challenges is ended.

A Trajectory Compensation Method Considering the Car-Following Behavior for Data Missing of Millimeter-Wave Radar in Roadside Detection Applications.

Concerning roadside traffic detection applications, and to address the millimeter-wave radar's missing data problem caused by target occlusion or the absence of features in low-speed conditions, this paper proposes a trajectory compensation method regarding car-following behavior. Referring to the installation scheme of the detector, a coordinate transformation method is presented to unify the radar spatial coordinates with the road coordinates. Considering the driver's car-following behavior, the optimal velocity model (OV), full velocity difference model (FVD), and the full velocity difference and acceleration (FVDA) model are applied for tracking the vehicle's trajectory related to the movement of the vehicle ahead. Finally, a data compensation scheme is presented. Taking actual trajectory data as samples, the proposed methods are verifiably useful for compensating for missing data and reconstructing target trajectories. Statistical results of different missing data trajectories demonstrate the rationality of the application of car-following models for the missing data compensation, and the FVDA model performs well compared with the OV and FVD models.

Biased-sample empirical likelihood weighting for missing data problems: an alternative to inverse probability weighting

AbstractInverse probability weighting (IPW) is widely used in many areas when data are subject to unrepresentativeness, missingness, or selection bias. An inevitable challenge with the use of IPW is that the IPW estimator can be remarkably unstable if some probabilities are very close to zero. To overcome this problem, at least three remedies have been developed in the literature: stabilizing, thresholding, and trimming. However, the final estimators are still IPW-type estimators, and inevitably inherit certain weaknesses of the naive IPW estimator: they may still be unstable or biased. We propose a biased-sample empirical likelihood weighting (ELW) method to serve the same general purpose as IPW, while completely overcoming the instability of IPW-type estimators by circumventing the use of inverse probabilities. The ELW weights are always well defined and easy to implement. We show theoretically that the ELW estimator is asymptotically normal and more efficient than the IPW estimator and its stabilized version for missing data problems. Our simulation results and a real data analysis indicate that the ELW estimator is shift-equivariant, nearly unbiased, and usually outperforms the IPW-type estimators in terms of mean square error.

A Randomization P-Value Test for Detecting Copying on Multiple-Choice Exams

This article is concerned with the statistical detection of copying on multiple-choice exams. As an alternative to existing permutation- and model-based copy-detection approaches, a simple randomization p-value (RP) test is proposed. The RP test, which is based on an intuitive match-score statistic, makes no assumptions about the distribution of examinees’ answer vectors and hence is broadly applicable. Especially important in this copy-detection setting, the RP test is shown to be exact in that its size is guaranteed to be no larger than a nominal α value. Additionally, simulation results suggest that the RP test is typically more powerful for copy detection than the existing approximate tests. The development of the RP test is based on the idea that the copy-detection problem can be recast as a causal inference and missing data problem. In particular, the observed data are viewed as a subset of a larger collection of potential values, or counterfactuals, and the null hypothesis of “no copying” is viewed as a “no causal effect” hypothesis and formally expressed in terms of constraints on potential variables.

Adjusting for publication bias in meta-analysis via inverse probability weighting using clinical trial registries.

Publication bias is a major concern in conducting systematic reviews and meta-analyses. Various sensitivity analysis or bias-correction methods have been developed based on selection models, and they have some advantages over the widely used trim-and-fill bias-correction method. However, likelihood methods based on selection models may have difficulty in obtaining precise estimates and reasonable confidence intervals, or require a rather complicated sensitivity analysis process. Herein, we develop a simple publication bias adjustment method by utilizing the information on conducted but still unpublished trials from clinical trial registries. We introduce an estimating equationfor parameter estimation in the selection function by regarding the publication bias issue as a missing data problem under the missing not at random assumption. With the estimated selection function, we introduce the inverse probability weighting (IPW) method to estimate the overall mean across studies. Furthermore, the IPW versions of heterogeneity measures such as the between-study variance and the I2 measure are proposed. We propose methods to construct confidence intervals based on asymptotic normal approximation as well as on parametric bootstrap. Through numerical experiments, we observed that the estimators successfully eliminated bias, and the confidence intervals had empirical coverage probabilities close to the nominal level. On the other hand, the confidence interval based on asymptotic normal approximation is much wider in some scenarios than the bootstrap confidence interval. Therefore, the latter is recommended for practicaluse.

Mining for equitable health: Assessing the impact of missing data in electronic health records

Electronic health records (EHR) are collected as a routine part of healthcare delivery, and have great potential to be utilized to improve patient health outcomes. They contain multiple years of health information to be leveraged for risk prediction, disease detection, and treatment evaluation. However, they do not have a consistent, standardized format across institutions, particularly in the United States, and can present significant analytical challenges– they contain multi-scale data from heterogeneous domains and include both structured and unstructured data. Data for individual patients are collected at irregular time intervals and with varying frequencies. In addition to the analytical challenges, EHR can reflect inequity– patients belonging to different groups will have differing amounts of data in their health records. Many of these issues can contribute to biased data collection. The consequence is that the data for under-served groups may be less informative partly due to more fragmented care, which can be viewed as a type of missing data problem. For EHR data in this complex form, there is currently no framework for introducing realistic missing values. There has also been little to no work in assessing the impact of missing data in EHR. In this work, we first introduce a terminology to define three levels of EHR data and then propose a novel framework for simulating realistic missing data scenarios in EHR to adequately assess their impact on predictive modeling. We incorporate the use of a medical knowledge graph to capture dependencies between medical events to create a more realistic missing data framework. In an intensive care unit setting, we found that missing data have greater negative impact on the performance of disease prediction models in groups that tend to have less access to healthcare, or seek less healthcare. We also found that the impact of missing data on disease prediction models is stronger when using the knowledge graph framework to introduce realistic missing values as opposed to random event removal.

Diagnosing and Handling Common Violations of Missing at Random

Ignorable likelihood (IL) approaches are often used to handle missing data when estimating a multivariate model, such as a structural equation model. In this case, the likelihood is based on all available data, and no model is specified for the missing data mechanism. Inference proceeds via maximum likelihood or Bayesian methods, including multiple imputation without auxiliary variables. Such IL approaches are valid under a missing at random (MAR) assumption. Rabe-Hesketh and Skrondal (Ignoring non-ignorable missingness. Presidential Address at the International Meeting of the Psychometric Society, Beijing, China, 2015; Psychometrika, 2023) consider a violation of MAR where a variable A can affect missingness of another variable B also when A is not observed. They show that this case can be handled by discarding more data before proceeding with IL approaches. This data-deletion approach is similar to the sequential estimation of Mohan et al. (in: Advances in neural information processing systems, 2013) based on their ordered factorization theorem but is preferable for parametric models. Which kind of data-deletion or ordered factorization to employ depends on the nature of the MAR violation. In this article, we therefore propose two diagnostic tests, a likelihood-ratio test for a heteroscedastic regression model and a kernel conditional independence test. We also develop a test-based estimator that first uses diagnostic tests to determine which MAR violation appears to be present and then proceeds with the corresponding data-deletion estimator. Simulations show that the test-based estimator outperforms IL when the missing data problem is severe and performs similarly otherwise.

Comparing the performance of eight imputation methods for propensity score matching in missing data problem

Propensity score (PS) is a popular method to control for covariates in observational studies. A challenge in PS analyses is missing values in covariates. This study aims to investigate how different imputation methods of handling missing values of covariates in a PS analysis can affect average treatment on the treated (ATT) estimates. In this study, missing data imputation methods were evaluated using different data sets, whose covariates were low, medium, and high (r=0.10, 0.50, 0.85) correlated with each other, for n=200 units and 1000 times running simulation. Missing data structures were created according to the missing at random (MAR) mechanism and different missing rates. Different datasets were obtained after having imputed the missing values separately by eight imputation methods including mean, median, mode, hot deck, last observation carried forward (LOCF), next observation carried backward (NOCB), regression and predictive mean matching (PMM). Then the PS nearest neighbor matching was implemented and ATT scores were obtained using the imputed data sets. The predictive performance of imputation methods was compared according to ATT scores by hierarchical cluster analysis with Euclidean distance complete linkage. ATT scores of regression and PMM methods were closer to each other and these methods showed the best predictive performance. Additionally, when there were larger amounts of missing data, the PMM was the best method of choice. Ignoring missing values on covariates for PS analyses causes information loss significantly and this information loss becomes greater as the rate of missing data increases. PS analyses might be biased if missing data on covariates are also ignored. To prevent this information loss and bias, PS analyses should be performed after solving the problem of missing data with MAR mechanism on covariates by regression and PMM methods, which showed statistical superiority compared to other methods in this study.

A Study of a Support Vector Machine Algorithm with an Orthogonal Legendre Kernel According to Neutrosophic logic and Inverse Lagrangian Interpolation

The decision-making process is greatly affected by the data collection stage. If the data collection process is not well controlled, i.e. there is some data lost due to the poor quality of the devices used or the lack of accuracy in the data entry process...etc., this will affect the work of the SVM algorithm, which is considered one of the best. Most of the workbooks suffer from the problems of missing and anomalous data. In this paper, we propose a method to treat the missing and anomalous data by reshaping the data set defined by the classical method into the neutrosophical data set by calculating the amount of true T, false F, and neutrality I in the neutrosophical set using inverse Lagrangian interpolation. We noticed the superiority of our proposed method for processing missing data over the method of [21], then we trained a support vector machine algorithm with orthogonal legender kernel on a breast cancer dataset taken from the Statistics Department of Al-Bayrouni Hospital in Damascus, where the proposed algorithm achieved a classification accuracy of 97%. The reason we chose a support vector machine classifier with an orthogonal legender kernel has two goals: the first is to eliminate the repetition of support vectors in the feature space. The second is to solve the problem of non-linear data distribution.

A Systematic Literature Review On Missing Values: Research Trends, Datasets, Methods and Frameworks

Handling of missing values in data analysis is the focus of attention in various research fields. Imputation is one method that is commonly used to overcome this problem of missing data. This systematic literature review research aims to present a comprehensive summary of the relevant scientific literature that describes the use of the imputation method in overcoming missing values. The literature search method is carried out using various academic databases and reliable sources of information. Relevant keywords are used to find articles that match the research question. After selection and evaluation, 40 relevant articles were included in this study. The findings of this study reveal a variety of imputation approaches and methods used in various research fields, such as social sciences, medicine, economics, and others. Commonly used imputation methods include single imputation, multivariate imputation, and model-based imputation methods. In addition, several studies also describe a combination of imputation methods to deal with more complex situations. The advantage of the imputation method is that it allows researchers to maintain sample sizes and minimize bias in data analysis. However, the research results also show that the imputation method must be applied with caution, because inappropriate imputation decisions can lead to biased results and can affect the accuracy of the research conclusions. In order to increase the validity and reliability of research results, researchers are expected to transparently report the imputation method used and describe the considerations made in the imputation decision-making process. This systematic review of the literature review provides an in-depth view of the use of the imputation method in handling missing values. In the face of the challenge of missing data, an understanding of the various imputation methods and the context in which they are applied will be key to generating meaningful findings in various research fields.

FRACTIONATION USING K MEANS CLUSTERING

The k-means algorithm is often used in clustering applications but its usage requires a complete data matrix. Missing data, however, is common in many applications. Mainstream approaches to clustering missing data reduce the missing data problem to a complete data formulation through either deletion or imputation but these solutions may incur significant costs. Our k-POD method presents a simple extension of k-means clustering for missing data that works even when the missingness mechanism is unknown, when external information is unavailable, and when there is significant missingness in the data.

Multi-objective deep learning framework for COVID-19 dataset problems

BackgroundIt has been reported that a deadly virus known as COVID-19 has arisen in China and has spread rapidly throughout the country. The globe was shattered, and a large number of people on the planet died. It quickly became an epidemic due to the absence of apparent symptoms and causes for patients, confusion appears due to the lack of sufficient laboratory results, and its intelligent algorithms were used to make decisions on clinical outcomes. MethodsThis study developed a new framework for medical datasets with high missing values based on deep-learning optimization models. The robustness of our model is achieved by combining: Data Missing Care (DMC) Framework to overcome the problem of high missing data in medical datasets, and Grid-Search optimization used to develop an improved deep predictive training model for patients with COVID-19 by setting multiple hyperparameters and tuning assessments on three deep learning algorithms: ANN (Artificial Neural Network), CNN (Convolutional Neural Network), and Recurrent Neural Networks (RNN). ResultsThe experiment results conducted on three medical datasets showed the effectiveness of our hybrid approach and an improvement in accuracy and efficiency since all the evaluation metrics were close to ideal for all deep learning classifiers. We got the best evaluation in terms of accuracy 98%, precession 98.5%, F1-score 98.6%, and ROC Curve (95% to 99%) for the COVID-19 dataset provided by GitHub. The second dataset is also Covid-19 provided by Albert Einstein Hospital with high missing data after applying our approach the accuracy reached more than 91%. Third dataset for Cervical Cancer provided by Kaggle all the evaluation metrics reached more than 95%. ConclusionsThe proposed formula for processing this type of data can replace the traditional formats in optimization while providing high accuracy and less time to classify patients. Whereas, the experimental results of our approach, supported by comprehensive statistical analysis, can improve the overall evaluation performance of the problem of classifying medical data sets with high missing values. Therefore, this approach can be used in many areas such as energy management, environment, and medicine.

Cost-Sensitive Laplacian Logistic Regression for Ship Detention Prediction

Port state control (PSC) is the last line of defense for substandard ships. During a PSC inspection, ship detention is the most severe result if the inspected ship is identified with critical deficiencies. Regarding the development of ship detention prediction models, this paper identifies two challenges: learning from imbalanced data and learning from unlabeled data. The first challenge, imbalanced data, arises from the fact that a minority of inspected ships were detained. The second challenge, unlabeled data, arises from the fact that in practice not all foreign visiting ships receive a formal PSC inspection, leading to a missing data problem. To address these two challenges, this paper adopts two machine learning paradigms: cost-sensitive learning and semi-supervised learning. Accordingly, we expand the traditional logistic regression (LR) model by introducing a cost parameter to consider the different misclassification costs of unbalanced classes and incorporating a graph regularization term to consider unlabeled data. Finally, we conduct extensive computational experiments to verify the superiority of the developed cost-sensitive semi-supervised learning framework in this paper. Computational results show that introducing a cost parameter into LR can improve the classification rate for substandard ships by almost 10%. In addition, the results show that considering unlabeled data in classification models can increase the classification rate for minority and majority classes by 1.33% and 5.93%, respectively.

Bayesian joint modeling of binomial and rank response with non-ignorable missing data for primate cognition

A random effects model for analyzing mixed rank and binomial data with considering the missing values is presented. Occurring of missing data is an important problem in all research fields. The most common approach to dealing with missing data is to delete cases containing missing observations. However, this approach reduces statistical power and mislead us to biased statistical results. This paper aims to prepare guidance for researchers facing missing data problems and to provide techniques for jointly modeling of binomial and rank responses. We compare the cognitive abilities of different primates based on their performance on 17 cognitive assessments obtained on either a rank or binomial scale using Bayesian latent variable with random effects models. Random effects are used to take into account the correlation between responses of the same individual.