Estimation Of Missing Values Research Articles

MethyLImp2: faster missing value estimation for DNA methylation data.

methyLImp, a method we recently introduced for the missing value estimation of DNA methylation data, has demonstrated competitive performance in data imputation compared to the existing, general-purpose, approaches. However, imputation running time was considerably long and unfeasible in case of large datasets with numerous missing values. methyLImp2 made possible computations that were previously unfeasible. We achieved this by introducing two important modifications that have significantly reduced the original running time without sacrificing prediction performance. First, we implemented a chromosome-wise parallel version of methyLImp. This parallelization reduced the runtime by several 10-fold in our experiments. Then, to handle large datasets, we also introduced a mini-batch approach that uses only a subset of the samples for the imputation. Thus, it further reduces the running time from days to hours or even minutes in large datasets. The R package methyLImp2 is under review for Bioconductor. It is currently freely available on Github https://github.com/annaplaksienko/methyLImp2.

A Probabilistic Framework for Missing Value Estimation in Multivariate IoT Data During Reefer Container Monitoring

A Probabilistic Framework for Missing Value Estimation in Multivariate IoT Data During Reefer Container Monitoring

Active Learning for Handling Missing Data.

Recently, the massive growth of IoT devices and Internet data, which are widely used in many applications, including industry and healthcare, has dramatically increased the amount of free unlabeled data collected. However, this unlabeled data is useless if we want to learn supervised machine learning models. The expensive and time-consuming cost of labeling makes the problem even more challenging. Here, the active learning (AL) technique provides a solution by labeling small but highly informative and representative data, which guarantees a high degree of generalizability over space and improves classification performance with data we have never seen before. The task is more difficult when the active learner has no predefined knowledge, such as initial training data, and when the obtained data is incomplete (i.e., contains missing values). In previous studies, the missing data should first be imputed. Then, the active learner selects from the available unlabeled data, regardless of whether the points were originally observed or imputed. However, selecting inaccurate imputed data points would negatively affect the active learner and prevent it from selecting informative and/or representative points, thus reducing the overall classification performance of the prediction models. This motivated us to introduce a novel query selection strategy that accounts for imputation uncertainty when querying new points. For this purpose, we first introduce a novel multiple imputation method that considers feature importance in selecting the most promising feature groups for missing values estimation. This multiple imputation method provides the ability to quantify the imputation uncertainty of each imputed data point. Furthermore, in each of the two phases of the proposed active learner (exploration and exploitation), imputation uncertainty is taken into account to reduce the probability of selecting points with high imputation uncertainty. We tested the effectiveness of the proposed active learner on different binary and multiclass datasets with different missing rates.

Opinion evolution and dynamic trust-driven consensus model in large-scale group decision-making under incomplete information

The shift to a new era of dealing with big data has driven continuous progress and development in computer science, artificial intelligence and machine learning. This change has led to the application of advanced techniques in the realm of decision science, particularly in the area of large-scale group decision-making (LSGDM). However, although these existing techniques have become the core of LSGDM methods, they are still limited in solving problems facing incomplete data. In addition, due to the rise of social media platforms such as Weibo, WeChat and Twitter, which build bridges for communication between decision makers (DMs), this brings new opportunities and challenges for consensus research. To address this set of issues, this study develops a consensus architecture that combines dynamic social network and opinion evolution in the context of an incomplete multi-attribute LSGDM. It is worth mentioning that the proposed consensus framework is a novel decision-making system that can be used to complete the estimation of the missing values and the consensus reaching process (CRP) by simulating the realistic decision-making scenarios. Firstly, considering the size of the trust value and the length of the path, a new trust propagation method is designed to achieve a more reliable estimation of the unknown trust value. Secondly, this paper establishes a missing value estimation method by virtue of the improved DeGroot model, which is able to obtain complete evaluation information by simulating the opinion formation process of DMs. Next, a hierarchical clustering algorithm with stronger robustness is constructed, which not only can adaptively complete the clustering process, but also integrally considers two attributes of trust and opinion similarity. In light of the above research, this study designs an opinion evolution and dynamic trust-driven consensus model, referred to as the DSN-DG-LSGDM model. Finally, the sensitivity analysis and experiments on a real dataset verify the significant superiority of the constructed DSN-DG-LSGDM model compared with the extant LSGDM consensus models.

Quantum mechanics-based missing value estimation framework for industrial data

As the importance of data-based predictive maintenance frameworks is rising, one of the emerging issues is the missing values in industrial data. Data with various missing values due to measurement sensor faults and power change issues can not only cause errors in future analyses but also lead to data shortages. Although various missing value estimation methods have been proposed to address this issue, correction methods that consider the fine uncertainty and noise of the missing values themselves have been relatively unexplored. Therefore, this study modeled missing values estimated using Gaussian progress regression by considering the fine movements and noises of data using a quantum mechanics-based stochastic differential equation and corrected them using Ito’s lemma. Estimating the missing data in this manner more closely simulates the attributes of data compared to existing estimation methods, enabling more accurate analysis. To demonstrate the excellence of the proposed framework, missing values were estimated using vehicle operation data and semiconductor manufacturing data with multiple missing values, and the estimation results were compared with those of existing algorithms.

Using Block Kriging as a Spatial Smooth Interpolator to Address Missing Values and Reduce Variability in Maize Field Yield Data

Block Kriging (a spatial interpolation method) and log10 transformation were compared for their effectiveness in reducing relative variance (coefficient of variance: CV) and estimate mean values in all harvested maize plants grown in three randomly taken field plots and for harvested plants after removing the “edge or margin” ones. The results showed that log10 transformation reduced CVs of all harvested original fresh weight (FW) plant data in the three plots from 35.6–41.6% (original data) to 6.0–7.5%, while the respective CVs due to Block Kriging were reduced to 14.5–19.9%. The back-log10-transformed means of all harvested FW plant data were reduced by 6.8–9.4%, while the respective reduction for plants excluding the margin ones was 1.3–8.3%. The Block Kriging means for all harvested FW plant data were reduced only by 0.3–0.4%, while the respective means of the harvested plants excluding margin ones were increased by 0.4–4.3%. These findings strongly suggest that Block Kriging should be preferred over the log10 transformation method (used so far by agroscientists) as it managed to effectively reduce variability in crop data and estimate missing values that provide more precise and reliable estimates of corn yield for farmers.

On the efficiency of convoluted weighted method for missing values in household surveys

Household surveys collect information on social and demographic characteristics in which the household constitutes the sampling units. They are effective means of obtaining variety of data needed for informed policy formulation and monitoring of national development. Despite their increasing recognition, household surveys suffer from non-response which creates bias estimates leading to wrong inference if not properly addressed. Existing estimators for handling missing values focused on the quality of missing value estimates without considering the resulting effects on location and scale parameters with regard to the nature of missingness and number of missing values. The convoluted weighted method was more efficient than the existing estimators with increasing number and nature of missingness. Its use will enhanced the level of precision of the estimated population parameter in the presence of missing values in household surveys.

MsImpute: Estimation of Missing Peptide Intensity Data in Label-Free Quantitative Mass Spectrometry

Mass spectrometry (MS) enables high-throughput identification and quantification of proteins in complex biological samples and can provide insights into the global function of biological systems. Label-free quantification is cost-effective and suitable for the analysis of human samples. Despite rapid developments in label-free data acquisition workflows, the number of proteins quantified across samples can be limited by technical and biological variability. This variation can result in missing values which can in turn challenge downstream data analysis tasks. General purpose or gene expression-specific imputation algorithms are widely used to improve data completeness. Here, we propose an imputation algorithm designated for label-free MS data that is aware of the type of missingness affecting data. On published datasets acquired by data-dependent and data-independent acquisition workflows with variable degrees of biological complexity, we demonstrate that the proposed missing value estimation procedure by barycenter computation competes closely with the state-of-the-art imputation algorithms in differential abundance tasks while outperforming them in the accuracy of variance estimates of the peptide abundance measurements, and better controls the false discovery rate in label-free MS experiments. The barycenter estimation procedure is implemented in the msImpute software package and is available from the Bioconductor repository.

A novel missing value imputation relying on K-means clustering and kernel-based weighting using grey relation (KWGI)

Data pre-processing is one of the crucial phases of data mining that enhances the efficiency of data mining techniques. One of the most important operations performed on data pre-processing is missing values imputation in incomplete datasets. This research presents a new imputation technique using K-means and samples weighting mechanism based on Grey relation (KWGI). The Grey-based K-means algorithm applicable to all samples of incomplete datasets clusters the similar samples, then an appropriate kernel function generates appropriate weights based on the Grey relation. The missing values estimation of the incomplete samples is done based on the weighted mean to reduce the impact of outlier and vague samples. In both clustering and imputation steps, a penalty mechanism has been considered to reduce the similarity of ambiguous samples with a high number of missing values, and consequently, increase the accuracy of clustering and imputation. The KWGI method has been applied on nine natural datasets with eight state-of-the-art and commonly used methods, namely CMIWD, KNNI, HotDeck, MeanI, KmeanI, RKmeanI, ICKmeanI, and FKMI. The imputation results are evaluated by the Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) criteria. In this study, the missing values are generated at two levels, namely sample and value, and the results are discussed in a wide range of missingness from low rate to high rate. Experimental results of the t-test show that the proposed method performs significantly better than all the other compared methods.

Category-aware optimal transport for incomplete data classification

Classification with incomplete data is practically challenging since missing data hinders classification. The two-stage strategy of filling missing values first and then classifying complete data degenerates performance due to the separate optimization of classification and imputation. While efforts have been made to classify incomplete data directly without imputation, the benefits of imputation on classification have been sacrificed. To this end, this paper proposes a Category-Aware Optimal Transport Neural Network (CAOT-NN) for incomplete data classification. Specifically, we design a category-aware optimal transport method conditioned on intra-category data distribution, which explicitly employs the category information of each sample and fills in discriminative values for the downstream classification module. Moreover, we reconstruct the observed values during the imputation process when performing classification learning, thus implicitly utilizing the category information of the classifier for missing values estimation. Extensive experiments conducted on 16 real-world and 30 synthetic datasets from the UCI repository demonstrate the superiority of CAOT-NN against existing state-of-the-art methods. Also, the visualization analysis reveals that CAOT-NN enables a tighter intra-category distribution of the imputed data, implying that the missing mechanism may be utilized as a data augmentation strategy to boost accuracy.

Unit consensus cost-based approach for group decision-making with incomplete probabilistic linguistic preference relations

In group decision-making, the incomplete probabilistic linguistic preference relation (InPLPR) has been widely studied because it can flexibly reflect the decision psychology of decision makers (DMs). In addition, to help DMs revise their opinions and reach a consensus, a large number of scholars have conducted research on the consensus reaching process. Therefore, this study proposes a group consensus decision approach based on InPLPR, considering consistency, the social trust network (STN), and unit cost consensus adjustment. First, an InPLPR consistency measure formula is proposed, based on which a multi-stage missing value estimation method based on optimal consistency and STN is offered for the missing information problem in InPLPR. Thereafter, the hesitation index and unit consensus cost measurement formulas are presented for incomplete probabilistic linguistic term set. Subsequently, consensus reaching and ranking selection are performed based on the proposed unit minimum cost consensus model. Finally, a numerical example of the future development choice problem for new energy vehicle enterprises is provided to demonstrate the applicability of the decision process, and the advantages of the method are derived through comparative analysis.

Formalizing Distributed Situation Awareness in Multi-Agent Networks

The version of distributed situation awareness (DSA) used in this article originated in human factors/ergonomics. Typically, this has involved the study of human operators working in teams and has used various forms of concept maps to qualitatively describe the information that team members use. In this article, we apply the concept of DSA to multi-agent teams (where team members might be human or automation) and extend the concept using the formal properties of Bayesian belief networks. In particular, we show how the Bayesian belief network can define DSA and how such a network can (using expectation maximization) adapt, even on the basis of limited information. The approach was considered in terms of situation awareness levels of perception (i.e., sensor state values conversion to belief) and projection (in terms of prediction and mission missing values estimation accuracy).

Uncertainty-Aware Variational-Recurrent Imputation Network for Clinical Time Series.

Electronic health records (EHR) consist of longitudinal clinical observations portrayed with sparsity, irregularity, and high dimensionality, which become major obstacles in drawing reliable downstream clinical outcomes. Although there exist great numbers of imputation methods to tackle these issues, most of them ignore correlated features, temporal dynamics, and entirely set aside the uncertainty. Since the missing value estimates involve the risk of being inaccurate, it is appropriate for the method to handle the less certain information differently than the reliable data. In that regard, we can use the uncertainties in estimating the missing values as the fidelity score to be further utilized to alleviate the risk of biased missing value estimates. In this work, we propose a novel variational-recurrent imputation network, which unifies an imputation and a prediction network by taking into account the correlated features, temporal dynamics, as well as uncertainty. Specifically, we leverage the deep generative model in the imputation, which is based on the distribution among variables, and a recurrent imputation network to exploit the temporal relations, in conjunction with utilization of the uncertainty. We validated the effectiveness of our proposed model on two publicly available real-world EHR datasets: 1) PhysioNet Challenge 2012 and 2) MIMIC-III, and compared the results with other competing state-of-the-art methods in the literature.

Building trait datasets: effect of methodological choice on a study of invasion

Trait-based approaches are commonly used to understand ecological phenomena and processes. Trait data are typically gathered by measuring local specimens, retrieving published records, or a combination of the two. Implications of methodological choices in trait-based ecological studies—including source of data, imputation technique, and species selection criteria—are poorly understood. We ask: do different approaches for dataset-building lead to meaningful differences in trait datasets? If so, do these differences influence findings of a trait-based examination of plant invasiveness, measured as abundance and spread rate? We collected on-site (Victoria, Australia) and off-site (TRY database) height and specific leaf area records for as many species as possible out of 157 exotic herbaceous plants. For each trait, we built six datasets of species-level means using records collected on-site, off-site, on-site and off-site combined, and off-site supplemented via imputation based on phylogeny and/or trait correlations. For both traits, the six datasets were weakly correlated (ρ = 0.31–0.95 for height; ρ = 0.14–0.88 for SLA), reflecting differences in species’ trait values from the various estimations. Inconsistencies in species’ trait means across datasets did not translate into large differences in trait-invasion relationships. Although we did not find that methodological choices for building trait datasets greatly affected ecological inference about local invasion processes, we nevertheless recommend: (1) using on-site records to answer local-scale ecological questions whenever possible, and (2) transparency around methodological decisions related to selection of study species and estimation of missing trait values.

Online subspace learning and imputation by Tensor-Ring decomposition

This paper considers the completion problem of a partially observed high-order streaming data, which is cast as an online low-rank tensor completion problem. Though the online low-rank tensor completion problem has drawn lots of attention in recent years, most of them are designed based on the traditional decomposition method, such as CP and Tucker. Inspired by the advantages of Tensor Ring decomposition over the traditional decompositions in expressing high-order data and its superiority in missing values estimation, this paper proposes two online subspace learning and imputation methods based on Tensor Ring decomposition. Specifically, we first propose an online Tensor Ring subspace learning and imputation model by formulating an exponentially weighted least squares with Frobenium norm regularization of TR-cores. Then, two commonly used optimization algorithms, i.e. alternating recursive least squares and stochastic-gradient algorithms, are developed to solve the proposed model. Numerical experiments show that the proposed methods are more effective to exploit the time-varying subspace in comparison with the conventional Tensor Ring completion methods. Besides, the proposed methods are demonstrated to be superior to obtain better results than state-of-the-art online methods in streaming data completion under varying missing ratios and noise.

Mechanism-aware imputation: a two-step approach in handling missing values in metabolomics

When analyzing large datasets from high-throughput technologies, researchers often encounter missing quantitative measurements, which are particularly frequent in metabolomics datasets. Metabolomics, the comprehensive profiling of metabolite abundances, are typically measured using mass spectrometry technologies that often introduce missingness via multiple mechanisms: (1) the metabolite signal may be smaller than the instrument limit of detection; (2) the conditions under which the data are collected and processed may lead to missing values; (3) missing values can be introduced randomly. Missingness resulting from mechanism (1) would be classified as Missing Not At Random (MNAR), that from mechanism (2) would be Missing At Random (MAR), and that from mechanism (3) would be classified as Missing Completely At Random (MCAR). Two common approaches for handling missing data are the following: (1) omit missing data from the analysis; (2) impute the missing values. Both approaches may introduce bias and reduce statistical power in downstream analyses such as testing metabolite associations with clinical variables. Further, standard imputation methods in metabolomics often ignore the mechanisms causing missingness and inaccurately estimate missing values within a data set. We propose a mechanism-aware imputation algorithm that leverages a two-step approach in imputing missing values. First, we use a random forest classifier to classify the missing mechanism for each missing value in the data set. Second, we impute each missing value using imputation algorithms that are specific to the predicted missingness mechanism (i.e., MAR/MCAR or MNAR). Using complete data, we conducted simulations, where we imposed different missingness patterns within the data and tested the performance of combinations of imputation algorithms. Our proposed algorithm provided imputations closer to the original data than those using only one imputation algorithm for all the missing values. Consequently, our two-step approach was able to reduce bias for improved downstream analyses.

A Method for Granular Traffic Data Imputation Based on PARATUCK2

Imputing missing data is a critical task in data-driven intelligent transportation systems. During recent decades there has been a considerable investment in developing various types of sensors and smart systems, including stationary devices (e.g., loop detectors) and floating vehicles equipped with global positioning system (GPS) trackers to collect large-scale traffic data. However, collected data may not include observations from all road segments in a traffic network for different reasons, including sensor failure, transmission error, and because GPS-equipped vehicles may not always travel through all road segments. The first step toward developing real-time traffic monitoring and disruption prediction models is to estimate missing values through a systematic data imputation process. Many of the existing data imputation methods are based on matrix completion techniques that utilize the inherent spatiotemporal characteristics of traffic data. However, these methods may not fully capture the clustered structure of the data. This paper addresses this issue by developing a novel data imputation method using PARATUCK2 decomposition. The proposed method captures both spatial and temporal information of traffic data and constructs a low-dimensional and clustered representation of traffic patterns. The identified spatiotemporal clusters are used to recover network traffic profiles and estimate missing values. The proposed method is implemented using traffic data in the road network of Manhattan in New York City. The performance of the proposed method is evaluated in comparison with two state-of-the-art benchmark methods. The outcomes indicate that the proposed method outperforms the existing state-of-the-art imputation methods in complex and large-scale traffic networks.

Research on Constructing Online Learning Performance Prediction Model Combining Feature Selection and Neural Network

Learning performance prediction can help teachers find students who tend to fail as early as possible so as to give them timely help, which is of great significance for online education. With the availability of online data and the continuous de-velopment of machine learning technology, learning performance prediction in large-scale online education is gaining new momentum. In order to understand the performance of different machine learning algorithms in predicting multi-category learning performance in large-scale online education, this study com-pares five machine learning algorithms including logic regression, decision tree algorithm, naive Bayes, support vector machine and deep neural network. Based on 100000 data records in the edX open dataset, this study models and forecasts students' learning performance with six online learning behaviors as the eigenval-ues. In the process of modeling, missing value estimation and normalization pre-processing are carried out on the original data at first. After that, six datasets of different sizes are divided as input data. Next, the performance of the five algo-rithms is tested on data sets of different sizes. Finally, SFS, SBS and multiple re-gression analysis are used to explore the effect of behavioral feature selection on algorithm performance. The research was validated and evaluated by three met-rics: precision, recall and F1 score. The results show that the F1 score of the deep neural network with multiple regression analysis feature selection achieves 99.25% in the large-scale dataset, outperforming the related other models by 1.25%.

A process monitoring and fault isolation framework based on variational autoencoders and branch and bound method

Nonlinear characteristic widely exists in industrial processes. Many approaches based on kernel methods and machine learning have been developed for nonlinear process monitoring. However, the fault isolation for nonlinear processes has rarely been studied in previous works. In this paper, a process monitoring and fault isolation framework is proposed for nonlinear processes using variational autoencoder (VAE) model. First, based on the probability graph model of VAE, a uniform monitoring index can be calculated by the probability density of observation variables. Then, the fault variables are estimated with normal variables by a missing value estimation method. The optimal fault variable set can be searched by branch and bound (BAB) algorithm. The proposed method can resolve the ”smearing effects” problem existing in traditional fault isolation methods. Finally, a numerical case and a hot strip mill process case are used to verified the proposed method.

Implication of climate variable selections on the uncertainty of reference crop evapotranspiration projections propagated from climate variables projections under climate change

The projection of reference crop evapotranspiration (ETo) based on global climate models (GCMs) is an important task in many fields under climate change background. However, ETo projections are typically hindered by the limited data availability or high uncertainty of the GCM-projected climate variable for ETo calculation (e.g., radiation and wind speed). In the current study, we investigate how the selection of climate variables for ETo calculation would influence the reliability and uncertainty of the ETo projections. Based on the Penman-Monteith (P-M) formula and the missing variable estimation approaches provided by FAO, five variable selection schemes (VSs) that select different subsets of the climate variables for ETo calculation are established. Four emission scenarios, five GCMs and four statistical downscaling approaches were adopted, and their uncertainty contributions were quantified by analyses of variation (ANOVA) approach. Results show that, among the climate variables except air temperature, shortwave radiation caused the lowest uncertainty to the projected ETo, and is thus suggested as the most suitable variable to be considered for ETo projections. In most areas, wind speed and relative humidity are determined as marginal variables for their limited influences in ETo projections, yet in some complex-terrain regions, wind speed could propagate considerable uncertainty to the projected ETo. Differences in GCMs is the major source of uncertainty in ETo projections, while the emission scenario generally ranked as second. Statistical downscaling approaches contribute limited uncertainty (below 10%) to ETo projections. Above all, this study shows the applicability of using the missing value estimation approaches for the FAO56 P-M formula to perform future ETo projection, for the practices with limited GCM data availability, and can serve as the reference for selecting suitable variables for ETo forecasting in applications.