Missing Data Values Research Articles

Improving McDiarmid Tree Performance for Predicting Heart Disease from Data Streams with Missing and Meaningless Values

Building machine learning algorithms for real-time clinical decision support systems has become a current research hotspot. The success of these algorithms depends on their ability to handle data stream characteristics. For example, we cite as characteristics the large amounts of data, the high speed and rate of incoming data, and the change in data nature and distribution over time. The Very Fast Decision Tree (VFDT) is a method for incrementally building decision trees. Since its proposition in the literature, it has become one of the most popular tools for data stream classification. This paper aims to optimize a new version of VFDT called McDiarmid Tree (MT) for the early prediction of heart diseases in real-time clinical decision support systems. The proposed method for improving MT performance consists of two main mechanisms: detecting the presence of missing and meaningless values in data attributes and handling the impact of this presence. The proposed MT has been compared with MT and VFDT. Simulation results show that the proposed MT attains significantly higher prediction accuracy with less time and model cost (RAM-Hours) than the other two algorithms.

GMA: Gap Imputing Algorithm for time series missing values

Data collected from the environment in computer engineering may include missing values due to various factors, such as lost readings from sensors caused by communication errors or power outages. Missing data can result in inaccurate analysis or even false alarms. It is therefore essential to identify missing values and correct them as accurately as possible to ensure the integrity of the analysis and the effectiveness of any decision-making based on the data. This paper presents a new approach, the Gap Imputing Algorithm (GMA), for imputing missing values in time series data. The Gap Imputing Algorithm (GMA) identifies sequences of missing values and determines the periodic time of the time series. Then, it searches for the most similar subsequence from historical data. Unlike previous work, GMA supports any type of time series and is resilient to consecutively missing values with different gaps distances. The experimental findings, which were based on both real-world and benchmark datasets, demonstrate that the GMA framework proposed in this study outperforms other methods in terms of accuracy. Specifically, our proposed method achieves an accuracy score that is 5 to 20% higher than that of other methods. Furthermore, the GMA framework is well suited to handling missing gaps with larger distances, and it produces more accurate imputations, particularly for datasets with strong periodic patterns.

Quantum mechanics-based missing value estimation framework for industrial data

As the importance of data-based predictive maintenance frameworks is rising, one of the emerging issues is the missing values in industrial data. Data with various missing values due to measurement sensor faults and power change issues can not only cause errors in future analyses but also lead to data shortages. Although various missing value estimation methods have been proposed to address this issue, correction methods that consider the fine uncertainty and noise of the missing values themselves have been relatively unexplored. Therefore, this study modeled missing values estimated using Gaussian progress regression by considering the fine movements and noises of data using a quantum mechanics-based stochastic differential equation and corrected them using Ito’s lemma. Estimating the missing data in this manner more closely simulates the attributes of data compared to existing estimation methods, enabling more accurate analysis. To demonstrate the excellence of the proposed framework, missing values were estimated using vehicle operation data and semiconductor manufacturing data with multiple missing values, and the estimation results were compared with those of existing algorithms.

Predicting Missing Value Data on IEC TC10 Datasets for Dissolved Gas Analysis using Tertius Algorithm

IEC TC10 is the most widely used Dissolved Gas Analysis (DGA) measurement dataset nowadays. Many DGA-based studies have been carried out using conventional methods and methods based on Artificial Intelligence Techniques (AITs). DGA is a diagnostic test performed on power transformers to detect and diagnose potential faults. The test involves analyzing the gases that are dissolved in the transformer oil, which can provide important information about the condition of the transformer. DGA is a widely used technique for transformer monitoring and maintenance in the power industry. However, this dataset is not perfect. There are still many problems in this dataset, one of which is the problem of missing value data. This problem will be significant if not appropriately handled. More reliable data from DGA measurement results is an in-dispensable reference in diagnosing faults in power transformers. This study focuses on dealing with the problem of missing value data using the Tertius algorithm, then testing the results using the J48 and Random Forest algorithms. The results obtained are pretty significant. Of the total 56 missing data, 36 could be predicted perfectly. And received the results of measuring accuracy using the J48 method of 62.73% and the Random Forest method of 70.71%. This result shows that the approach we applied is relatively good for handling missing values in IEC TC10 datasets.

Bayesian joint modelling of life expectancy and healthy life expectancy and valuation of retirement village contract

Life expectancy and healthy life expectancy are both important indicators of population wellbeing, reflecting the expected length of life and duraiton of healthy living respectively. While extensive research has focused on projecting life expectancy, very few have investigated the projection and simulation of healthy life expectancy and examined its potential applications. In this paper, we establish a Bayesian approach to jointly model life expectancy and healthy life expectancy. We apply this approach to the population data of various countries, demonstrating its ability to provide accurate forecasts, generate probability intervals to capture future uncertainty, and estimate missing data values. We then illustrate an application of the Bayesian model in valuing a retirement village contract. Many Australian seniors choose retirement villages for the community support they offer, transitioning to aged care facilities as their health declines. The value of the accommodation service depends on the resident’s length of stay, which can be approximated by healthy life expectancy. The proposed Bayesian model can produce a sampled distribution of the contract’s net present value, providing a useful risk management tool for retirement village operators. Our findings emphasise the importance of considering healthy life expectancy in policies on public health and retirement.

Transformer Fault Diagnosis Method Based on Incomplete Data and TPE-XGBoost

Dissolved gas analysis is an important method for diagnosing the operating condition of power transformers. Traditional methods such as IEC Ratios and Duval Triangles and Pentagon methods are not applicable in the case of abnormal or missing values of DGA data. A novel transformer fault diagnosis method based on an extreme gradient boosting algorithm is proposed in this paper. First, the traditional statistical method is replaced by the random forest regression algorithm for filling in missing values of dissolved gas data. Normalization and feature derivation of the outlier data is adopted based on the gas content. Then, hyperparameter optimization of the transformer fault diagnosis model based on an extreme gradient boosting algorithm is carried out using the tree-structured probability density estimator algorithm. Finally, the influence of missing data and optimization algorithms on transformer fault diagnosis models is analyzed. The effects of different algorithms based on incomplete datasets are also discussed. The results show that the performance of the random forest regression algorithm on missing data filling is better than classification and regression trees and traditional statistical methods. The average accuracy of the fault diagnosis method proposed in the paper is 89.5%, even when the missing data rate reaches 20%. The accuracy and robustness of the TPE-XGBoost model are superior to other machine learning algorithms described in this paper, such as k-nearest neighbor, deep neural networks, random forest, etc.

#3940 RISK OF MAJOR ADVERSE CARDIOVASCULAR EVENTS ACCORDING TO DIALYSIS MODALITY

Abstract Background and Aims Among dialysis patients, cardiovascular events are the leading cause of death. Little is known about difference in the frequency of cardiovascular events between various dialysis modalities. Glucose load may contribute to a metabolic burden in peritoneal dialysis (PD). On the other hand, hemodialysis can cause intradialytic hypotension, cardiac stunning, and arrhythmias, which are associated with increased risk of death. We compared risk of major cardiovascular events in patients who started continuous ambulatory PD (CAPD), automated PD (APD) and home HD with in-center HD patients. Method We included 968 patients who entered dialysis in the Helsinki-Uusimaa healthcare district in Finland from 2004 to 2017, of whom 162 were on CAPD, 229 on APD, 145 on home HD and 432 on in-center HD at day 90 from the start of dialysis. Patients were followed up for 5 years or until the end of 2019. Major adverse cardiovascular event (MACE) was defined as acute myocardial infarction, stroke or death due to cardiovascular disease. The cumulative incidence of the first MACE was calculated by taking other causes of death into account as competing risk events and censoring at time of kidney transplantation. Cox regression was used to compare risk of MACE between dialysis modalities with adjustment for age, gender, primary renal disease, prior comorbidities (coronary heart disease, left ventricular hypertrophy, heart failure and stroke), and laboratory data (plasma albumin, phosphate and ionized calcium). Imputation was performed to replace missing values of comorbidities and laboratory data. Results Of all 968 patients, 195 (20%) experienced a MACE during the entire follow-up and 62 (6.4%) during the first year of follow-up. The cumulative incidence of first MACE was similar in in-center HD and CAPD patients and higher than that in APD and home HD patients (Figure 1). Without adjustments, the hazard ratio of MACE was 0.83 [95% CI 0.56–1.2] for CAPD, 0.49 [95% CI 0.31–0.77] for APD and 0.42 [95% CI 0.23–0.78] for home HD in comparison to in-center HD. After adjustment for possible confounders, the hazard ratio of MACE was 1.1 [95% CI 0.70–1.6] for CAPD, 0.88 [95% CI 0.53–1.5] for APD and 0.80 [95% CI 0.41–1.6] for home HD and not statistically significantly different in comparison to in-center HD. Conclusion We observed a lower risk of MACE among patients who entered APD or home hemodialysis compared to in-center hemodialysis, but after adjusting for potential confounding factors, the difference diminished and was no longer statistically significant.

Research on multi-context aware recommendation methods based on tensor factorization

Compared to the traditional recommender systems, context-aware recommender systems are more in line with actual application contexts. However, the existing researches are mostly focused on single context-aware recommendation, such as time-aware recommendation or location-aware recommendation, and lack of in-depth research on multi-context-aware recommendation. Therefore, we proposed a recommendation method of high-order tensor factorization based on multi-context-aware. First, on the basis of analyzing the influence of context on users’ interest preferences, the sensitivity of users to multiple contexts was detected using statistical methods. For context-sensitive users, four-dimensional tensors and feature matrices used to solve data sparsity were constructed based on rating matrix and situational information. And then the stochastic gradient descent algorithm was used for iterative calculation to fill in missing data values and carry out parameter optimization. For context-insensitive users, we used matrix factorization to predict users’ interest preferences. Finally, we tested and validated our method on a multi-context-aware movie dataset, and the experimental results show that the proposed method could effectively reduce the prediction error and improve the recommendation quality.

A meta heuristic optimization algorithm (Lion-BES-XGB) for water quality prediction

ABSTRACT Recently, water contamination has become a major problem in developing countries due to urbanization and population growth, leading to an increase in morbidity and mortality rates.Therefore, accurate water quality prediction is crucial in the urban water supply system. In this work, we developed a prediction model based on Extreme Gradient Boosting (XGB) using a hybrid feature selection approach combining Lion Swarm Optimization (LSO) and Bald Eagle Search (BES). The proposed method LSO-BES-XGB consists of three steps: preprocessing, feature selection, and classification.Z-score normalization helps fill in missing data values by scaling to indicate the number of standard deviations from the mean. LSO-BES Feature selection identifies relevant features, and the XGB classifier determines whether the water is normal or contaminated. The LSO-BES-XGB model was applied to the Cauvery River data set and achieved 94.22% accuracy, 93.12% precision, 94.23% recall, and 92.45%.

Accurate and interpretable gene expression imputation on scRNA-seq data using IGSimpute.

Single-cell ribonucleic acid sequencing (scRNA-seq) enables the quantification of gene expression at the transcriptomic level with single-cell resolution, enhancing our understanding of cellular heterogeneity. However, the excessive missing values present in scRNA-seq data hinder downstream analysis. While numerous imputation methods have been proposed to recover scRNA-seq data, high imputation performance often comes with low or no interpretability. Here, we present IGSimpute, an accurate and interpretable imputation method for recovering missing values in scRNA-seq data with an interpretable instance-wise gene selection layer (GSL). IGSimpute outperforms 12 other state-of-the-art imputation methods on 13 out of 17 datasets from different scRNA-seq technologies with the lowest mean squared error as the chosen benchmark metric. We demonstrate that IGSimpute can give unbiased estimates of the missing values compared to other methods, regardless of whether the average gene expression values are small or large. Clustering results of imputed profiles show that IGSimpute offers statistically significant improvement over other imputation methods. By taking the heart-and-aorta and the limb muscle tissues as examples, we show that IGSimpute can also denoise gene expression profiles by removing outlier entries with unexpectedly high expression values via the instance-wise GSL. We also show that genes selected by the instance-wise GSL could indicate the age of B cells from bladder fat tissue of the Tabula Muris Senis atlas. IGSimpute can impute one million cells using 64min, and thus applicable to large datasets.

Estimating the Values of Missing Data Related to Infrastructure Condition States Using Their Spatial Correlation

Infrastructure managers consistently monitor the condition of their assets to predict their deterioration speed and determine the optimal time to execute preventive interventions. However, despite the recent progress in more frequent and accurate monitoring of assets and storage of the related results, in practice, real-world data often contains errors and discrepancies such as missing data or faulty entries. This problem can happen owing to collection errors during routine inspections or inconsistency of data storage formats in different years. Because the quality of data plays a significant role in the accuracy of deterioration prediction and the resulting intervention programs, it is important to improve condition state predictions by imputing the values of missing information. This paper examines the efficiency of different models that use the spatial correlation of infrastructure assets in predicting the value of missing data. The models include univariate and multivariate Kriging, a hybrid artificial neural network (ANN)-Kriging model, and the bidirectional long short term memory (bi-LSTM) neural network, which can model the data with spatial correlation or a sequential relationship. The results confirm that the condition indicator values can be estimated with reasonably low levels of error.

Imputation of Missing Values in Time Series Using an Adaptive-Learned Median-Filled Deep Autoencoder.

Missing values are ubiquitous in industrial data sets because of multisampling rates, sensor faults, and transmission failures. The incomplete data obstruct the effective use of data and degrade the performance of data-driven models. Numerous imputation algorithms have been proposed to deal with missing values, primarily based on supervised learning, that is, imputing the missing values by constructing a prediction model with the remaining complete data. They have limited performance when the amount of incomplete data is overwhelming. Moreover, many methods have not considered the autocorrelation of time-series data. Thus, an adaptive-learned median-filled deep autoencoder (AM-DAE) is proposed in this study, aiming to impute missing values of industrial time-series data in an unsupervised manner. It continuously replaces the missing values by the median of the input data and its reconstruction, which allows the imputation information to be transmitted with the training process. In addition, an adaptive learning strategy is adopted to guide the AM-DAE paying more attention to the reconstruction learning of nonmissing values or missing values in different iteration periods. Finally, two industrial examples are used to verify the superior performance of the proposed method compared with other advanced techniques.

Novel MIA-LSTM Deep Learning Hybrid Model with Data Preprocessing for Forecasting of PM2.5

Day by day pollution in cities is increasing due to urbanization. One of the biggest challenges posed by the rapid migration of inhabitants into cities is increased air pollution. Sustainable Development Goal 11 indicates that 99 percent of the world’s urban population breathes polluted air. In such a trend of urbanization, predicting the concentrations of pollutants in advance is very important. Predictions of pollutants would help city administrations to take timely measures for ensuring Sustainable Development Goal 11. In data engineering, imputation and the removal of outliers are very important steps prior to forecasting the concentration of air pollutants. For pollution and meteorological data, missing values and outliers are critical problems that need to be addressed. This paper proposes a novel method called multiple iterative imputation using autoencoder-based long short-term memory (MIA-LSTM) which uses iterative imputation using an extra tree regressor as an estimator for the missing values in multivariate data followed by an LSTM autoencoder for the detection and removal of outliers present in the dataset. The preprocessed data were given to a multivariate LSTM for forecasting PM2.5 concentration. This paper also presents the effect of removing outliers and missing values from the dataset as well as the effect of imputing missing values in the process of forecasting the concentrations of air pollutants. The proposed method provides better results for forecasting with a root mean square error (RMSE) value of 9.8883. The obtained results were compared with the traditional gated recurrent unit (GRU), 1D convolutional neural network (CNN), and long short-term memory (LSTM) approaches for a dataset of the Aotizhonhxin area of Beijing in China. Similar results were observed for another two locations in China and one location in India. The results obtained show that imputation and outlier/anomaly removal improve the accuracy of air pollution forecasting.

Spatial–Temporal Data Imputation Model of Traffic Passenger Flow Based on Grid Division

Traffic flows (e.g., the traffic of vehicles, passengers, and bikes) aim to reveal traffic flow phenomena generated by traffic participants in traffic activities. Various studies of traffic flows rely heavily on high-quality traffic data. The taxi GPS trajectory data are location data that include latitude, longitude, and time. These data are critical for traffic flow analysis, planning, infrastructure layout, and recommendations for urban residents. A city map can be divided into multiple grids according to the latitude and longitude coordinates, and traffic passenger flows data derived from taxi trajectory data can be extracted. However, random missing data occur due to weather and equipment failure. Therefore, the effective imputation of missing traffic flow data is a hot topic. This study proposes the spatio-temporal generative adversarial imputation net (ST-GAIN) model to solve the traffic passenger flows imputation. An adversarial game with multiple generators and one discriminator is established. The generator observes some components of the time-domain and regional traffic data vector extracted from the grid. It effectively imputes the missing values of the spatio-temporal traffic passenger flow data. The experimental data are accurate Kunming taxi trajectory data, and experimental results show that the proposed method outperforms five baseline methods regarding the imputation accuracy. It is significant and suggests the possibility of effectively applying the model to predict the passenger flows in some areas where traffic data cannot be collected for some reason or traffic data are randomly missing.

Expanding Tidy Data Principles to Facilitate Missing Data Exploration, Visualization and Assessment of Imputations

Despite the large body of research on missing value distributions and imputation, there is comparatively little literature with a focus on how to make it easy to handle, explore, and impute missing values in data. This paper addresses this gap. The new methodology builds upon tidy data principles, with the goal of integrating missing value handling as a key part of data analysis workflows. We define a new data structure, and a suite of new operations. Together, these provide a connected framework for handling, exploring, and imputing missing values. These methods are available in the R package naniar.

Novel Methods for Imputing Missing Values in Water Level Monitoring Data

Hydrological data are collected automatically from remote water level monitoring stations and then transmitted to the national water management centre via telemetry system. However, the data received at the centre can be incomplete or anomalous due to some issues with the instruments such as power and sensor failures. Usually, the detected anomalies or missing data are just simply eliminated from the data, which could lead to inaccurate analysis or even false alarms. Therefore, it is very helpful to identify missing values and correct them as accurate as possible. In this paper, we introduced a new approach - Full Subsequence Matching (FSM), for imputing missing values in telemetry water level data. The FSM firstly identifies a sequence of missing values and replaces them with some constant values to create a dummy complete sequence. Then, searching for the most similar subsequence from the historical data. Finally, the identified subsequence will be adapted to fit the missing part based on their similarity. The imputation accuracy of the FSM was evaluated with telemetry water level data and compared to some well-established methods - Interpolation, k-NN, MissForest, and also a leading deep learning method - the Long Short-Term Memory (LSTM) technique. Experimental results show that the FSM technique can produce more precise imputations, particularly for those with strong periodic patterns.

ANN in forecasting Missing Rainfall Data

ANN has been used to estimate rainfall data by analysing patterns and mapping the correlation between historical data and weather patterns. ANNs are a type of machine learning algorithm that is modelled on the structure of the human brain, which makes them particularly effective for solving complex problems that involve large amounts of data. Overall, the use of ANNs for estimating rainfall data is a promising area of research that has the potential to provide valuable insights into weather patterns and their impacts on the environment and human society. The potential of ANN in the estimation of missing precipitation values has been investigated in detail. This study proposes to compare the performance of conventional methods as well as different algorithms of the Artificial Neural Network method to predict missing rainfall in the Upper Tapi catchment area in the West region of India. It was found that the ANN method has an edge over the conventional methods and proved to be a better method of finding the missing rainfall data values.

A Comparison of a Radial Basis Function Neural Network with other Methods for Estimating Missing Values in Univariate Time Series

Missing values in the time-series data set have an impact on the correct decision-making in the future. Since complete data helps to obtain high accuracy in the estimation process, the reason for missing values is a malfunction of the measuring device or an error in the data entry process by the person. The research aims to compare the radial basis function methods with other methods to estimate missing values in univariate time series data. the simulation method was used to compare the methods to estimate the missing values, and that was used the Box-Jenkins model AR(1) once with the value of and once with the value of and with different sample sizes (60,100,300), assuming four percentages of missing from data values are missing at random MAR (5%,10%,15%,20%). The accuracy of the estimation of the methods was evaluated by using the standard of accuracy, the mean sum of squares error (MSE). from the results obtained using simulation, it was found that the(RBF) method is the best method for estimating the missing values in both values, in all sizes, and all loss ratios because it produces the lowest value of the average square error compared to other methods.
 
 Paper type: Research paper

NEPSscaling: plausible value estimation for competence tests administered in the German National Educational Panel Study

Educational large-scale assessments (LSAs) often provide plausible values for the administered competence tests to facilitate the estimation of population effects. This requires the specification of a background model that is appropriate for the specific research question. Because the German National Educational Panel Study (NEPS) is an ongoing longitudinal LSA, the range of potential research questions and, thus, the number of potential background variables for the plausible value estimation grow with each new assessment wave. To facilitate the estimation of plausible values for data users of the NEPS, the R package NEPSscaling allows their estimation following the scaling standards in the NEPS without requiring in-depth psychometric expertise in item response theory. The package requires the user to prepare the data for the background model only. Then, the appropriate item response model including the linking approach adopted for the NEPS is selected automatically, while a nested multiple imputation scheme based on the chained equation approach handles missing values in the background data. For novice users, a graphical interface is provided that only requires minimal knowledge of the R language. Thus, NEPSscaling can be used to estimate cross-sectional and longitudinally linked plausible values for all major competence assessments in the NEPS.

STA-GAN: A Spatio-Temporal Attention Generative Adversarial Network for Missing Value Imputation in Satellite Data

Satellite data is of high importance for ocean environment monitoring and protection. However, due to the missing values in satellite data, caused by various force majeure factors such as cloud cover, bad weather and sensor failure, the quality of satellite data is reduced greatly, which hinders the applications of satellite data in practice. Therefore, a variety of methods have been proposed to conduct missing data imputation for satellite data to improve its quality. However, these methods cannot well learn the short-term temporal dependence and dynamic spatial dependence in satellite data, resulting in bad imputation performance when the data missing rate is large. To address this issue, we propose the Spatio-Temporal Attention Generative Adversarial Network (STA-GAN) for missing value imputation in satellite data. First, we develop the Spatio-Temporal Attention (STA) mechanism based on Graph Attention Network (GAT) to learn features for capturing both short-term temporal dependence and dynamic spatial dependence in satellite data. Then, the learned features from STA are fused to enrich the spatio-temporal information for training the generator and discriminator of STA-GAN. Finally, we use the generated imputation data by the trained generator of STA-GAN to fill the missing values in satellite data. Experimental results on real datasets show that STA-GAN largely outperforms the baseline data imputation methods, especially for filling satellite data with large missing rates.