Missing Value Prediction Research Articles

Enhancing water quality prediction for fluctuating missing data scenarios: A dynamic Bayesian network-based processing system to monitor cyanobacteria proliferation

Tackling the impact of missing data in water management is crucial to ensure the reliability of scientific research that informs decision-making processes in public health. The goal of this study is to ascertain the root causes associated with cyanobacteria proliferation under major missing data scenarios. For this purpose, a dynamic missing data management methodology is proposed using Bayesian Machine Learning for accurate surface water quality prediction of a river from Limia basin (Spain). The methodology used entails a sequence of analytical steps, starting with data pre-processing, followed by the selection of a reliable dynamic Bayesian missing value prediction system, leading finally to a supervised analysis of the behavioral patterns exhibited by cyanobacteria. For that, a total of 2,118,844 data points were used, with 205,316 (9.69 %) missing values identified. The machine learning testing showed the iterative structural expectation maximization (SEM) as the best performing algorithm, above the dynamic imputation (DI) and entropy-based dynamic imputation methods (EBDI), enhancing in some cases the accuracy of imputations by approximately 50 % in R2, RMSE, NRMSE, and logarithmic loss values. These findings can impact how data on water quality is being processed and studied, thus, opening the door for more reliable water management strategies that better inform public health decisions.

A novel model for rainfall prediction using hybrid stochastic-based Bayesian optimization algorithm.

Rainfall forecasting is considered one of the key concerns in the meteorological department because it is related strongly to social as well as economic factors. But, because of modern context of climatic conditions and the intense activities of humans, the forecasting procedure of rainfall patterns becomes more problematic. Therefore, this paper proposes a novel timely and reliable rainfall prediction model using a hybrid stochastic Bayesian optimization approach (HS-BOA). The weather dataset containing different meteorological geographical features is provided as input to the introduced prediction method. Hybrid stochastic (HS) specifications are tuned by the Bayesian optimization algorithm (BOA) to upgrade the prediction accuracy. The weather data are initially preprocessed through the pipelines, namely, data separation, missing value prediction, weather condition cod separation, and normalization. After preprocessing, the highly correlated features are removed by correlation matrix using the Pearson correlation coefficient. Then, the most significant features which contribute more to predicting rainfall are selected through the feature selection process. At last, the suggested rainfall forecasting model accurately predicts rainfall using optimized parameters. The experimental analysis is performed, and for the proposed HS-BOA, MAE, RMSE, and COD, values attained for rainfall prediction are 0.513mm, 59.90mm, and 40.56mm respectively. As a result, the proposed HS-BOA approach achieves minimum error rates with increased prediction accuracy than other existing approaches.

LSH-based missing value prediction for abnormal traffic sensors with privacy protection in edge computing

Traffic flow prediction is an important part of intelligent transportation systems (ITS). However, sensor failures or the transmission distortion often occur in the process of data acquisition, which will inevitably cause the loss or abnormality of traffic flow data transmitted to the edge server. In this situation, it is necessary to share traffic flow data among different platforms. However, existing traffic flow prediction methods are facing two challenges in the process of traffic flow data sharing. First, user privacy is often leaked in the process of sharing traffic data on various platforms. Moreover, with the continuous updating of data, the efficiency and scalability of data sharing between different platforms will become lower and lower. In view of the above challenges, in this paper, we propose a novel prediction method for the missing traffic flow data caused by abnormal sensors, named ASMVP_{distr-LSH} based on distributed locality-sensitive hashing (LSH) technique. At last, a case study is presented to illustrate the feasibility and effectiveness of our approach ASMVP_{distr-LSH}.

Swarm Intelligence in Internet of Medical Things: A Review.

Continuous advancements of technologies such as machine-to-machine interactions and big data analysis have led to the internet of things (IoT) making information sharing and smart decision-making possible using everyday devices. On the other hand, swarm intelligence (SI) algorithms seek to establish constructive interaction among agents regardless of their intelligence level. In SI algorithms, multiple individuals run simultaneously and possibly in a cooperative manner to address complex nonlinear problems. In this paper, the application of SI algorithms in IoT is investigated with a special focus on the internet of medical things (IoMT). The role of wearable devices in IoMT is briefly reviewed. Existing works on applications of SI in addressing IoMT problems are discussed. Possible problems include disease prediction, data encryption, missing values prediction, resource allocation, network routing, and hardware failure management. Finally, research perspectives and future trends are outlined.

Urban traffic flow prediction: a dynamic temporal graph network considering missing values

Accurate traffic flow prediction on the urban road network is an indispensable function of Intelligent Transportation Systems (ITS), which is of great significance for urban traffic planning. However, the current traffic flow prediction methods still face many challenges, such as missing values and dynamic spatial relationships in traffic flow. In this study, a dynamic temporal graph neural network considering missing values (D-TGNM) is proposed for traffic flow prediction. First, inspired by the Bidirectional Encoder Representations from Transformers (BERT), we extend the classic BERT model, called Traffic BERT, to learn the dynamic spatial associations on the road structure. Second, we propose a temporal graph neural network considering missing values (TGNM) to mine traffic flow patterns in missing data scenarios for traffic flow prediction. Finally, the proposed D-TGNM model can be obtained by integrating the dynamic spatial associations learned by Traffic BERT into the TGNM model. To train the D-TGNM model, we design a novel loss function, which considers the missing values problem and prediction problem in traffic flow, to optimize the proposed model. The proposed model was validated on an actual traffic dataset collected in Wuhan, China. Experimental results showed that D-TGNM achieved good prediction results under four missing data scenarios (15% random missing, 15% block missing, 30% random missing, and 30% block missing), and outperformed ten existing state-of-the-art baselines.

Usage of Clustering and Weighted Nearest Neighbors for Efficient Missing Data Imputation of Microarray Gene Expression Dataset

AbstractA complete dataset is essential for most bioinformatics analytical techniques, including gene expression data categorization, prognosis, and prediction. Due to sensor malfunction, software inability, or human error, the gene sample value may be missing. In gene expression experiments, missing data has a massive effect on analyzing the data obtained. Consequently, this has become a crucial issue requiring an efficient imputation technique to address. This research provided a technique for predicting missing values by using clustering and top K closest neighbor techniques that consider the local similarity pattern. The K‐means method is integrated with a spectral clustering methodology. After optimizing the clustering parameters, cluster size, and weighting criteria, missed gene sample values are estimated. The top K closest neighbor method uses weighted distance to predict the missed gene sample value falling in a specific cluster. Experimental outcomes show that the suggested imputation methodology generates efficient predictions compared to existing imputation techniques. In this research, microarray datasets comprising information from various cancers and tumors are used to experiment with the imputation performance. The primary contribution of this work is that even if the microarray dataset has varied dimensions and features, local similarity‐based approaches may be employed for missing value prediction.

Improved Collaborative Filtering Recommender System Based on Missing Values Imputation on E-Commerce

One of the important aspects in e-commerce is how to recommend a product to users accurately. To achieve this goal, many e-commerce starts to build and research about recommender system. Many methods can be used to build a recommender system, one of them is using the collaborative filtering technique. This technique often experiences data sparsity problem that can impact to the recommender system prediction accuracy. To solve this problem, we apply improved collaborative filtering. This method predicts the missing values in the user item rating matrix. First, we do an initial selection to determine potential users who have the same characteristics with the active user. After that, we calculate the average distance between the active user and the other selected user. Next, we calculate missing values prediction. Missing values predictions is only done for items that have never been rated by other’s selected user but has been rated by the active user. We used Amazon electronic product with high sparsity level in this research to simulate the actual condition of e-commerce. We used MAE and RMSE to measure prediction accuracy. The methods we apply succeeds to improve the prediction accuracy compare to the conventional collaborative filtering method. The average MAE for method that we apply is 0.78 and RMSE 1.07

Hybrid Missing Value Imputation Algorithm- KLR

The data mining process mainly deals with the estimation, prediction, pattern extraction, and classification in big databases. The presence of missing values in the dataset decreases the accuracy of data mining classifiers. Therefore it is necessary to deal with missing values in the dataset to achieve accurate results. To improve the quality of data and prediction accuracy in the classification process, the authors have proposed a new hybrid missing value prediction algorithm, KLR, by combining the KNN and linear regression approach. The proposed KLR algorithm has been used for class validation and missing values imputation. Wisconsin Breast Cancer Diagnostic Dataset of 569 instances with 32 attributes from the machine learning repository of UCI, Irvinewasused to conduct the study. The Pearson Coefficient Correlation method is used for feature selection.Data normalization is performed using Min-max scaling technique. The Scikit-learn library for machine learning in python is used to complete all the experiments as the experimental framework. The mean square error method is used to evaluate the performance of the model. The proposed KLR algorithm with 450 nearest neighbors out of 569 gives the lowest MSE ie 0.00188 and more accurately predicts the missing values as compared to the classic models.

West Nile Virus Prediction Based on Data Mining

This paper performed some exploratory data visualization on this data set. The nature and representation of input data was found out and the preliminary feature selection was conducted in this step. And this paper performed data preprocessing and feature engineering on this data set, which had critical importance of the accuracy of prediction results. The paper built multiple regression models on the missing values prediction in the testing set. The paper implemented various data mining algorithms to build predictive models, including Gaussian Naive Bayes classifier, K-Nearest Neighbors (K-NN) algorithm, Multi-layer Perceptron (MLP), Logistic regression, random forest and XGBoost. After the experiments, XGBoost classifier could give the best result among all the models.

Evaluation of Missing Value Estimation for Microarray Data

Publisher: School of Statistics, Renmin University of China, Journal: Journal of Data Science, Title: Evaluation of Missing Value Estimation for Microarray Data, Authors: Danh V. Nguyen, Naisyin Wang, Raymond J. Carroll

Enhance the Quality of Collaborative Filtering Using Tagging

Background: Due to enormous data on web sites, recommending users for every item is impossible. For this problem, Recommender Systems (RS) are introduced. RS is categorized into Content-Based (CB), Collaborative Filtering (CF) and Hybrid RS. Based on these techniques, recommendations are made to the user. In this, CF is the recent technique used in RS in which tagging features also provided. Objective: Three main issues occur in RS are scalability problem which occurs when there is a huge data, sparsity problem occurs when rating data is missing and cols start user or item problem occurs when new user or new item enters in the system. To avoid these issues, here we have proposed Tag and Time weight model with GA in Collaborative Tagging. Methods: Here we have proposed a method Collaborative Tagging (CT) with Tag and Time weight model with real value genetic algorithm which enhances the recommendation quality by removing the issues of sparsity and cold start user problems with the help of missing value prediction. Here in this system, the sparsity problem can be removed using missing value prediction and cold start problems are removed using tag and time weight model using GA. Results: In this study, we have compared the results of Collaborative Filtering with Cosine Similarity (CF-CS), Collaborative Filtering with Diffusion Similarity (CF-DS), Tag and Time Weight Model with Diffusion Similarity (TAW-TIW-DS) and Tag and Time Weight Model Using Diffusion Similarity and Genetic Algorithm (TAW-TIW-DS-GA). Conclusion: Here we have compared the proposed approach with the baseline approaches and the metrics are used MAE, prediction percentage, Hit-rate and Hit-rank. Based on these metrics for every split, TAW-TIW-DS-GA showed best results as compared to the existing approach.

Effective Context-Aware Recommendations Based on Context Weighting Using Genetic Algorithm and Alleviating Data Sparsity

ABSTRACT Context-aware collaborative filtering (CACF) is an effective approach for adapting recommendations under users’ specific contextual situations and aims to improve predictive accuracy for Context-aware recommender systems (CARSs). Incorporating context in recommender systems (RSs) considering the equal importance to all contextual dimensions is not appropriate for seeking an intelligent and useful recommendation. In this paper, we propose a Real-coded Genetic Algorithm (RCGA) based CARS framework that exploits contextual pre-filtering and contextual modeling paradigms into CACF with appropriate context feature weights for enhancing accuracy as well as the diversity of the recommendation list. Further to alleviate the data sparsity, an effective missing value prediction (EMVP) algorithm is applied into proposed framework. The accuracy based on RCGA is compared with other two schemes: Support Vector Machine (SVM) and Particle Swarm Optimization (PSO), and RCGA has shown better results. Experimental results based on real-world datasets have clearly established the effectiveness of our proposed CARS schemes.

Feature embedding and conditional neural processes for data imputation

This Letter addresses the missing value prediction problem in data. Existing missing value prediction studies apply their algorithms after replacing missing values with a constant value, which does not represent the missing values clearly and has disadvantages in processing data with a high missing rate. The authors propose a model that generalises a single data sample from its observations and predicts the value of a query feature. The authors' model can deal with an incomplete data sample independently of missing values. In this process, the feature embedding matrix and feature importance weights are introduced to represent each feature and learned jointly with the proposed model. Through experiments, they verify that the proposed model effectively predicts the missing values of data.

Efficient pattern matching for uncertain time series data with optimal sampling and dimensionality reduction

Time series data mining becomes an active research area due to the rapid proliferation of temporal-dependent applications. Dimensionality reduction and uncertainty handling play a pivotal role in extracting the time series pattern. Most of the dimensionality reduction schemes are designed based on the assumption that every class of samples follows the Gaussian distribution. Lack of this property in real time data distribution does not allow dimensionality reduction techniques to characterize the different classes well and measure the data uncertainty accurately. In addition to, applying an uncertainty measurement evenly on inconsistent time series data samples may underestimate the source of uncertainty among various sub-samples. This paper presents the Handling UNcertainty and missing value prediction in Time series (HUNT). The proposed approach employs Adaptive Reservoir Filling for sampling the time series and Discrepant Sample dependent Chebyshev inequality for handling the uncertainty. The HUNT implements the adaptive reservoir filling using discrepancy estimation over a statistical population and decides the reservoir size according to the variations in the data stream. The state of the statistical population ensures the uncertainty handling over discrepant samples. The proposed approach precisely replaces the missing values with the support of the Mean-Mode imputation method. To effectively select the key features, it applies both the indirect and direct performance measures on the statistical samples. Finally, the proposed model generates the fine-tuned statistical samples through segmentation to facilitate the time series pattern matching. The experimental results demonstrate that the HUNT approach significantly outperforms the existing time series pattern matching approaches such as KSample approach by 18% higher recall and UG-Miner approach by 20% minimum Mean Absolute Error (MAE) while testing on the Weather forecasting dataset.

MOWM: Multiple Overlapping Window Method for RBF based missing value prediction on big data

The problem of missing values in the process of data acquisition is becoming critical due to both hardware failure and human error. Radial Basis Function based interpolation and missing value prediction in the dependent variable through surface fitting has been the viable solution from a long time. However, the solution works well on the idea of building one equation of ‘N’ weight variables corresponding to each sampled data point from a set of ‘N’ samples. Still, because of the inherent computations, it demands big primary memory when ‘N’ becomes large. Hence, on a memory-restricted setup, sometimes it suffers from the memory overflow problem. In this paper, we propose a novel data decomposition based RBF enabled surface fitting approach by building and aggregating multiple small models in an overlapping manner, which works well even with smaller primary memory with minimal impact on loss of generality. We also consider two hyperparameters as window size and overlapping index in order to tune bias-variance tradeoff. The proposed approach is applied to ten real-world datasets having multiple dimensions and results are found competitive while comparing with single trained model and Kernel Ridge Regression. Therefore, we believe this approach will rejuvenate the RBF based surface fitting method in attaining better performance in the Big data world too.

Missing Values Prediction for Cyber Vulnerability Analysis in Academic Institutions

Missing Values Prediction for Cyber Vulnerability Analysis in Academic Institutions

Mining intrinsic information by matrix factorization-based approaches for collaborative filtering in recommender systems

Matrix factorization (MF) is an increasingly important approach in the field of missing value prediction because recommender systems are rapidly becoming ubiquitous. MF-based collaborative filtering (CF) seeks to improve recommender performance by combining user–item matrix with MF. However, most MF-based approaches available at present could not obtain high prediction accuracy because of the sparse availability of user–item matrices in CF models. The present paper proposes a framework that involves two efficient MF, dynamic single-element-based CF-integrating manifold regularization (DSMMF) and dynamic single-element-based Tikhonov graph regularization non-negative MF (DSTNMF). The aim of this framework is to better use the intrinsic structure of user–item rating matrix and user/item content information, overcome the dimensionality curse and ill-posed problem of weighted graph NMF, and evade the frequent manipulations of indicator matrices that lack practicability. We validate the effectiveness of our proposed algorithms with respect to recommender performance by four indices on three datasets. We demonstrate that our proposed approaches lead to considerable improvement compared with several other state-of-the-art approaches.

Imputation of missing genetic markers SNP using linear regression models

For imputation of missing SNP are used softwares which require known relationship between genotyped individuals. In common breeding business the genotypes of parents are not always known. That is why our own methodological process was used. The aim of this study is to map the current research of genetic chips and to verify the calculation process. The testing was processed at chosen loci in two datasets and in 8 models with different amount of SNPs. For the dataset A was prediction of missing values almost accurate with model reliability 100% with the exception of one homozygous locus where the reliability reached only 55%. In the dataset B the most extensive model reached the reliability of 80 - 90% even in case of homozygous loci. The prediction error value was higher than in the first case. It was proven that missing values prediction is possible to calculate using the neighbouring SNPs.

A novel distance-based iterative sequential KNN algorithm for estimation of missing values in microarray gene expression data

The presence of missing entries in DNA microarray gene expression datasets creates severe problems in downstream analysis because they require complete datasets. Though several missing value prediction methods have been proposed to solve this problem, they have limitations which may affect the performance of various analysis algorithms. In this regard, a novel distance based iterative sequential K-nearest neighbour imputation method ISKNNimpute has been proposed. The proposed distance is a hybridisation of modified Euclidean distance and Pearson correlation coefficient. The proposed method is a modification of KNN estimation in which the concept of reuse of estimation is considered using both iterative and sequential approach. The performance of the proposed ISKNNimpute method is tested on various time-series and non time-series microarray datasets comparing with several widely used existing imputation techniques. The experimental results confirm that the ISKNNimpute method consistently generates better results compared to other existing methods.

Predicting Web Service QoS via Combining Matrix Factorization with Network Location

With the increasing abundance of Web Services across Internet, Quality of Service (QoS)based service recommendation has become a hot issue. It is necessary to predict the missing values of QoS for service recommendation. Because Web services run on the Internet, their network locations may be anther critical factor for QoS prediction. Although there have existed many works on QoS prediction, few consider the influence of the network locations of users or Web Services. In this paper, we propose a novel collaborative QoS prediction framework with network location-based regularization (NLBR). We first elaborate the popular Matrix Factorization (MF) model for missing values prediction. Then, by taking advantage of the local connectivity between Web services users, we incorporate network location information to identify the neighborhood. We conduct the experiments on a public large-scale real-world QoS dataset, Experiments show that our proposed approaches have the better prediction performance compared with the existed approaches.