Missing Value Imputation Research Articles

MResGat: Multi-head Residual Dilated Convolution Assisted Gated Unit Framework for Crop Yield Prediction

The importance of predicting crop yields has increased due to growing concerns of surrounding food security. Early forecasting of crop yields holds a pivotal role in avoiding starvations by estimating the food supply available for the expanding global population. Several Deep Learning (DL) and Machine Learning (ML) techniques are involved to develop effective and accurate crop yield prediction model. Nevertheless, existing models faces some limitations such as less accuracy, high error rate because of noisy data, high training time and extracted less effective features for prediction. To overcome these issues, the novel DL methodology is introduced for attaining high accurate crop yield prediction. Initially, the soil, weather and other resources big data are collected from the various agriculture field. In data collection phase, the input data of larger size are stored in the Hadoop platform for the purpose of storing as well as processing the entire data in a distributed manner. The data are pre-processed through the utilization of Missing value imputation and z-score based data normalization. From the pre-processed data, the optimal features are considered using Integrated Correlation Random recursive elimination (InCorRe) approach. Based on the previous soil and weather information, the suitable yield of crops can be predicted using Multi head Residual dilated convolution assisted gated unit (MResGat) model. Finally, the losses of the network model can be optimized using African vulture optimization algorithm (AVO). The proposed method is evaluated using the several performance metrics, which achieved 0.023% of MSE value and 0.036% of MAE values.

Implementing Stacking with Cross - Validation for Heart Disease Prediction.

The objective of this work is to apply machine learning techniques for the prediction and early identification of cardiovascular disease, a major worldwide health problem. XGBoost, K-nearest neighbors (KNN), decision tree (DT), support vector machine (SVM), and StackingCVClassifier were among the ensemble algorithms used to anticipate cardiac disease using a dataset that came from the UCI ML database. To improve the quality of the data, an exploratory data analysis was performed on the dataset using methods including outlier identification and missing value imputation. Stacking CV Classifier attained the best accuracy rate of 92.20%, according to a comparative examination of pre-processing and post-processing findings. When compared to previous approaches, the suggested strategies performed better in terms of accuracy, recall, and f1-score. Moreover, the flexibility of the model indicates its possible application to other illnesses with comparable features.

Analysis and Visualization of Quantitative Proteomics Data Using FragPipe-Analyst.

The FragPipe computational proteomics platform is gaining widespread popularity among the proteomics research community because of its fast processing speed and user-friendly graphical interface. Although FragPipe produces well-formatted output tables that are ready for analysis, there is still a need for an easy-to-use and user-friendly downstream statistical analysis and visualization tool. FragPipe-Analyst addresses this need by providing an R shiny web server to assist FragPipe users in conducting downstream analyses of the resulting quantitative proteomics data. It supports major quantification workflows, including label-free quantification, tandem mass tags, and data-independent acquisition. FragPipe-Analyst offers a range of useful functionalities, such as various missing value imputation options, data quality control, unsupervised clustering, differential expression (DE) analysis using Limma, and gene ontology and pathway enrichment analysis using Enrichr. To support advanced analysis and customized visualizations, we also developed FragPipeAnalystR, an R package encompassing all FragPipe-Analyst functionalities that is extended to support site-specific analysis of post-translational modifications (PTMs). FragPipe-Analyst and FragPipeAnalystR are both open-source and freely available.

Prediction of oil production based on multivariate analysis and self-attention mechanism integrated with long short-term memory

The accurate oil prediction of wells is essential for making informed decisions regarding the extension of the well lifespan and the enhancement of oil recovery rates. However, the prediction of oil well production is highly challenging due to the complex, nonlinear, and non-stationary data influenced by reservoir geological characteristics and operational adjustments. To address this, a novel prediction method for oil well production is proposed in this study. Firstly, the data-cleaning approach designed in this study is utilized to eliminate outliers from the raw dataset and impute missing values. Subsequently, relevant features are identified through analysis to form a usable dataset. The proposed deep learning neural network, namely the self-attention mechanism integrated with long short-term memory, is trained and learned on this dataset. Finally, the predictive performance of the model is validated using a set of actual oil production data. Through data experiments, the proposed model effectively predicts oil well production with superior accuracy compared to baseline models, achieving an R-squared of 0.872. This method provides reliable decision support for optimizing oil field development and management.

Predicting suicidal ideation from irregular and incomplete time series of questionnaires in a smartphone-based suicide prevention platform: a pilot study

Over 700,000 people die by suicide annually. Collecting longitudinal fine-grained data about at-risk individuals, as they occur in the real world, can enhance our understanding of the temporal dynamics of suicide risk, leading to better identification of those in need of immediate intervention. Self-assessment questionnaires were collected over time from 89 at-risk individuals using the EMMA smartphone application. An artificial intelligence (AI) model was trained to assess current level of suicidal ideation (SI), an early indicator of the suicide risk, and to predict its progression in the following days. A key challenge was the unevenly spaced and incomplete nature of the time series data. To address this, the AI was built on a missing value imputation algorithm. The AI successfully distinguished high SI levels from low SI levels both on the current day (AUC = 0.804, F1 = 0.625, MCC = 0.459) and three days in advance (AUC = 0.769, F1 = 0.576, MCC = 0.386). Besides past SI levels, the most significant questions were related to psychological pain, well-being, agitation, emotional tension, and protective factors such as contacts with relatives and leisure activities. This represents a promising step towards early AI-based suicide risk prediction using a smartphone application.

The performance of prognostic models depended on the choice of missing value imputation algorithm: a simulation study

ObjectiveThe development of clinical prediction models is often impeded by the occurrence of missing values in the predictors. Various methods for imputing missing values before modelling have been proposed. Some of them are based on variants of multiple imputation by chained equations, while others are based on single imputation. These methods may include elements of flexible modelling or machine learning algorithms, and for some of them user-friendly software packages are available. The aim of this study was to investigate by simulation if some of these methods consistently outperform others in performance measures of clinical prediction models. Study Design and SettingWe simulated development and validation cohorts by mimicking observed distributions of predictors and outcome variable of a real data set. In the development cohorts, missing predictor values were created in 36 scenarios defined by the missingness mechanism and proportion of non-complete cases. We applied three imputation algorithms that were available in R software: mice, aregImpute and missForest. These algorithms differed in their use of linear or flexible models, or random forests, the way of sampling from the predictive posterior distribution, and the generation of a single or multiple imputed data sets. For multiple imputation we also investigated the impact of the number of imputations. Logistic regression models were fitted with the simulated development cohorts before (full data analysis) and after missing value generation (complete case analysis), and with the imputed data. Prognostic model performance was measured by the scaled Brier score, c-statistic, calibration intercept and slope, and by the mean absolute prediction error evaluated in validation cohorts without missing values. Performance of full data analysis was considered as ideal. ResultsNone of the imputation methods achieved the model’s predictive accuracy that would be obtained in case of no missingness. In general, complete case analysis yielded the worst performance, and deviation from ideal performance increased with increasing percentage of missingness and decreasing sample size. Across all scenarios and performance measures, aregImpute and mice, both with 100 imputations, resulted in highest predictive accuracy. Surprisingly aregImpute outperformed full data analysis in achieving calibration slopes very close to 1 across all scenarios and outcome models. The increase of mice’s performance with 100 compared to 5 imputations was only marginal. The differences between the imputation methods decreased with increasing sample sizes and decreasing proportion of non-complete cases. ConclusionIn our simulation study, model calibration was more affected by the choice of the imputation method than model discrimination. While differences in model performance after using imputation methods were generally small, multiple imputation methods as mice and aregImpute that can handle linear or nonlinear associations between predictors and outcome are an attractive and reliable choice in most situations.

A novel and efficient risk minimisation-based missing value imputation algorithm

Missing value imputation (MVI) is a key task in data science, in which learning models are built from incomplete data. In contrast to externally driven MVI algorithms, this study proposes a novel risk minimisation-based MVI algorithm (RM-MVI) that considers both the internal characteristics of missing data and the external performance for specific classification applications. RM-MVI is technically designed for labelled data and is applied in two stages: filling with structural risk minimisation (SRM) and refining with empirical risk minimisation (ERM). In the filling stage, an autoencoder with a single hidden layer is trained on the original dataset without missing values. Missing values are first initialised with random numbers, and the imputation values are then preliminarily optimised based on the derived updating rule to minimise the structural risk-oriented objective function. After the imputation values have been preliminarily optimised in the filling stage, a neural-network-based classifier is trained in the refining stage to optimise the imputation values sophisticatedly by reducing the empirical risk. Experiments were conducted on several benchmark datasets to validate the feasibility, rationality, and effectiveness of the proposed RM-MVI algorithm. The results show that (1) the optimisation processes of the imputation values corresponding to the SRM and ERM are convergent so that the optimised imputation values can be obtained; (2) SRM can ensure distribution consistency of the imputation values that are preliminarily optimised in the filling stage, while ERM can optimise the imputation values sophisticatedly in the refining stage, which is more helpful for classifier training; and (3) the RM-MVI algorithm can yield considerably better MVI performance on benchmark datasets than 11 well-known MVI algorithms, such as a 26% higher distribution consistency ratio and 2% to 5% higher testing accuracies for 6 classifiers on average. This demonstrates that RM-MVI is a viable approach for addressing MVI problems.

Data Discretization Impact on Deep Learning for Missing Value Imputation of Continuous Data

Data Discretization Impact on Deep Learning for Missing Value Imputation of Continuous Data

Integrated ECOD-KNN Algorithm for Missing Values Imputation in Datasets: Outlier Removal

Missing data cause the incompleteness of data sets and can lead to poor performance of models which also can result in poor decisions, despite using the best handling methods. When there is a presence of outliers in the data, using KNN algorithm for missing values imputation produce less accurate results. Outliers are anomalies from the observations and removing outliers is one of the most important pre-processing step in all data analysis models. KNN algorithms are able to adapt to missing value imputation even though they are sensitive to outliers, which might end up affecting the quality of the imputation results. KNN is mainly used among other machine learning algorithms because it is simple to implement and have a relatively high accuracy. In the literature, various studies have explored the application of KNN in different domains, however failing to address the issue of how sensitive it is to outliers. In the proposed model, outliers are identified using a combination of the Empirical- Cumulative-distribution-based Outlier Detection (ECOD), Local Outlier Factor (LOF) and isolation forest (IForest). The outliers are substituted using the median of the non- outlier data and the imputation of missing values is done using the k-nearest neighbors algorithm. For the evaluation of the model, different metrics were used such as the Root Mean Square Error (RMSE), (MSE), R2 squared (R2 ) and Mean Absolute Error (MAE). It clearly indicated that dealing with outliers first before imputing missing values produces better imputation results than just using the traditional KNN technique which is sensitive to outliers.

An Enhanced Adaptive Medical Diagnostic Model Driven By Genetic–Neuro-Fuzzy Algorithms

This research aims at developing adaptive medical diagnostic system driven by genetic, neural network and fuzzy logic algorithms for effective diagnosis of tuberculosis. Medical diagnosis for tuberculosis disease is a complex decision process that involves a lot of vagueness and uncertainty management, since the disease has multiple symptoms. The use of several algorithms has been explored in clinical diagnosis models for diagnosing and prescribing therapy for tuberculosis, but has challenges in transforming the vagueness and uncertainty of multiple symptoms due to the noisy nature of the disease data and over fitting of the models, this had led to the challenge of accurate classification, training, optimization, diagnosis and prescription of therapy. Object Oriented Analysis and Design (OOAD) and System Structured Analysis and Design (SSAD) methodologies were adopted in the research. The methodology involved obtaining four hundred and thirty (430) clinical data from patients with tuberculosis records at the Tuberculosis and Leprosy Hospital, Eku, Delta State. The obtained data were pre-processed using missing values imputation and numeric data encoding methods. Thereafter, a genetic algorithm was applied to the processed data to select six relevant features from the initial number of 16 features (Cough, Night sweats, Fever, Systolic Blood Pressure, Difficulty in Breathing, Loss of appetite, Sputum, Chills, Loss of pleasure, Immune Suppression, Chest Pain, Lack of concentration, Irritation, Loss of energy, Lymph Node Enlargement, Body Mass Index). The reduced dataset was split into training and test sets. The ANFIS (Adaptive Neuro-Fuzzy Inference System) was applied to perform diagnosis in which it was trained and validated on training and test sets respectively. The ANFIS was driven by Mamdani’s inference mechanism with sixty-four (64) generated rules with confidence and support scores of above 10% and 15% respectively. The developed model performance on the test sets gave an accuracy of 90% precision, sensitivity, and specificity of 100%. The results showed that all the attributes of tuberculosis contributed to the degree of the diseases based on their respective weights. Conclusively, the system accurately classified, trained attributes, and predicts the severity of tuberculosis.

Autoreplicative random forests with applications to missing value imputation

Autoreplicative random forests with applications to missing value imputation

A novel framework to generate plant functional groups for ecological modelling

An effective way to reduce complexity in ecological modelling is by grouping species that share similar characteristics into functional groups or types. Often, the creation of plant functional groups (PFGs) is carried out for each case study in an ad-hoc way using a small set of traits. This limits the transferability of these PFGs to other geographical areas or study systems. We propose a novel generic framework to generate PFGs that considers the most important ecological dimensions, is applicable to case studies globally, and that emerges from patterns of functional redundancy across species. Based on most relevant and measured plant characteristics, we designed a multi-step process that includes: i) data harmonisation and missing values imputation; ii) species clustering based on multiple characteristics encompassing the main ecological dimensions featured in plant community ecological models (i.e., dispersal, competition, and demography) and iii) the combination of ecological dimension-specific groups into comprehensive PFGs. We demonstrate this framework by applying it to a global dataset of plant characteristics including a functional traits dataset and a plant-soil co-occurrence dataset for 19,102 species. Lastly, to test the ability of generated PFGs to summarise species’ functional variation within plant communities, we correlate taxonomical and functional diversity indices calculated at the species and at the PFGs level across a global dataset of plant communities (sPlotOpen). Our framework generated 465 global, robust data-driven PFGs with non-overlapping combinations of traits for each ecological dimension divided by growth form. The validation returned positive correlation values between PFGs and species-level diversity metrics, supporting the ability of the obtained PFGs to capture functional and taxonomic diversity patterns across a variety of plant communities worldwide. The framework allows for the easy integration of newly available species characteristics data. The obtained global PFGs, covering all main known ecological processes and environmental conditions at small resolution, can increase the predictive power and accuracy of process-based models and help furthering varying-scale ecological studies.

A novel data-characteristic-driven modeling approach for imputing missing value in industrial statistics: A case study of China electricity statistics

As a direct reference tool to reflect the operational status and development level of national industry, industrial statistics hold significant value for numerous systematic studies. Nevertheless, it is crucial to recognize that the quality of these statistics can be compromised by the common occurrence of missing value. This issue poses a substantial challenge for analyzing and utilizing industrial statistics, impeding progress in tasks reliant upon them. Given the severity of the missing value problem in industrial statistical databases and the limitations of existing literatures on missing value imputation in terms of research objects and modeling approaches, this paper proposes a novel missing value imputation modeling approach for single-indicator panels of industrial statistics based on the idea of data-characteristic-driven (DCD). Accordingly, taking the inter-provincial “monthly power generation” data from China as an example, the imputation model was constructed and its validity was tested under different imputed objects (Jiangsu and Jilin), different missing types (continuous and discrete), and different missing rates (5%, 10% and 20%) respectively. The results indicate that the proposed DCD modeling approach in this paper exhibits excellent efficacy. The imputation model, constructed based on the data characteristic of the imputed object, demonstrates clear advantages in handling missing value with different missing types and rates. This is evident in its superior consideration of numerical accuracy, directional accuracy, and imputation stability, resulting in an outstanding comprehensive imputation effect.

Development and evaluation of interpretable machine learning regressors for predicting femoral neck bone mineral density in elderly men using NHANES data.

Osteoporotic femoral neck fractures (OFNFs) pose a significant orthopedic challenge in the elderly population, accounting for up to 40% of all osteoporotic fractures and leading to considerable health deterioration and increased mortality. In addressing the critical need for early identification of osteoporosis through routine screening of femoral neck bone mineral density (FNBMD), this study developed a user-friendly prediction model aimed at men aged 50 years and older, a demographic often overlooked in osteoporosis screening. Utilizing data from the National Health and Nutrition Examination Survey (NHANES), the study involved outlier detection and handling, missing value imputation via the K nearest neighbor (KNN) algorithm, and data normalization and encoding. The dataset was split into training and test sets with a 7:3 ratio, followed by feature screening through the least absolute shrinkage and selection operator (LASSO) and the Boruta algorithm. Eight different machine learning algorithms were then employed to construct predictive models, with their performance evaluated through a comprehensive metric suite. The random forest regressor (RFR) emerged as the most effective model, characterized by key predictors such as age, body mass index (BMI), poverty income ratio (PIR), serum calcium, and race, achieving a coefficient of determination (R²) of 0.218 and maintaining robustness in sensitivity analyses. Notably, excluding race from the model resulted in sustained high performance, underscoring the model's adaptability. Interpretations using Shapley additive explanations (SHAP) highlighted the influence of each feature on FNBMD. These findings indicate that our predictive model effectively aids in the early detection of osteoporosis, potentially reducing the incidence of OFNFs in this high-risk population.

Iterative missing value imputation based on feature importance

Iterative missing value imputation based on feature importance

Comparing preprocessing strategies for 3D-Gene microarray data of extracellular vesicle-derived miRNAs

BackgroundExtracellular vesicle-derived (EV)-miRNAs have potential to serve as biomarkers for the diagnosis of various diseases. miRNA microarrays are widely used to quantify circulating EV-miRNA levels, and the preprocessing of miRNA microarray data is critical for analytical accuracy and reliability. Thus, although microarray data have been used in various studies, the effects of preprocessing have not been studied for Toray’s 3D-Gene chip, a widely used measurement method. We aimed to evaluate batch effect, missing value imputation accuracy, and the influence of preprocessing on measured values in 18 different preprocessing pipelines for EV-miRNA microarray data from two cohorts with amyotrophic lateral sclerosis using 3D-Gene technology.ResultsEighteen different pipelines with different types and orders of missing value completion and normalization were used to preprocess the 3D-Gene microarray EV-miRNA data. Notable results were suppressed in the batch effects in all pipelines using the batch effect correction method ComBat. Furthermore, pipelines utilizing missForest for missing value imputation showed high agreement with measured values. In contrast, imputation using constant values for missing data exhibited low agreement.ConclusionsThis study highlights the importance of selecting the appropriate preprocessing strategy for EV-miRNA microarray data when using 3D-Gene technology. These findings emphasize the importance of validating preprocessing approaches, particularly in the context of batch effect correction and missing value imputation, for reliably analyzing data in biomarker discovery and disease research.

Revisiting COVID-19’s Impact on Cycling: An Examination of Bicycle Count Data in Montréal

During the earlier stages of the COVID-19 pandemic, numerous studies used bicycle count data to understand the pandemic’s impact on urban cycling. However, given the context and timeliness of such studies, correcting for periods of missing data and ensuring data accuracy were not always possible. In this paper, we examine the quality of bicycle count data, using Montreal as a case study, and propose methodological improvements. By employing a missing value imputation technique and excluding sensors potentially affected by exogenous factors, we hope to provide cities with a more precise assessment of cycling trends.

Optimized Extreme Gradient Boosting with Remora Algorithm for Congestion Prediction in Transport Layer

Transmission control protocol (TCP) is the most common protocol found in recent networks to maintain reliable communication. The most popular transport protocol in use today is TCP that cannot fully utilize the ability of the network because of the constraints of its conservative congestion control algorithm and favor’s reliability over timeliness. Despite congestion is the most frequent cause of lost packets, transmission defects can also result in packet loss. In response to packet loss, end-to-end congestion control mechanism in TCP limits the amount of remarkable, unacknowledged data segments that are permitted in the network. To overcome the drawback, Optimized Extreme Gradient Boosting Algorithm is proposed to predict the congestion. Initially, the data is collected and given to data preprocessing to improve the data quality. Min-Max normalization is used to normalize the data in the particular range and KNN-based missing value imputation is used to replace the missing values in the original data in the preprocessing section. Then the preprocessed data is fed into the Optimized Extreme Gradient Boosting Algorithm to predict the congestion. Remora optimization is used in the designed model for optimally selecting the learning rate to minimize the error for enhancing the prediction accuracy in machine learning. For validating the proposed model, the performance metrics attained by the proposed and existing model are compared. Accuracy, precision, recall and error values for the proposed methods are 96%, 97%, 96% and 3% values are obtained. Thus, the proposed optimized extreme gradient boosting with the remora algorithm for congestion prediction in the transport layer method is the best method than the existing algorithm.

Attention-based Imputation of Missing Values in Electronic Health Records Tabular Data.

The imputation of missing values (IMV) in electronic health records tabular data is crucial to enable machine learning for patient-specific predictive modeling. While IMV methods are developed in biostatistics and recently in machine learning, deep learning-based solutions have shown limited success in learning tabular data. This paper proposes a novel attention-based missing value imputation framework that learns to reconstruct data with missing values leveraging between-feature (self-attention) or between-sample attentions. We adopt data manipulation methods used in contrastive learning to improve the generalization of the trained imputation model. The proposed self-attention imputation method outperforms state-of-the-art statistical and machine learning-based (decision-tree) imputation methods, reducing the normalized root mean squared error by 18.4% to 74.7% on five tabular data sets and 52.6% to 82.6% on two electronic health records data sets. The proposed attention-based missing value imputation method shows superior performance across a wide range of missingness (10% to 50%) when the values are missing completely at random.

Alleviating cold start and sparsity problems in the micro, small, and medium enterprises marketplace using clustering and imputation techniques

Recommendation systems are often implemented in e-commerce and micro, small, and medium enterprises (MSMEs) marketplaces to improve consumer services by providing product recommendations according to their interests. However, it still faces problems, namely sparsity and cold start, thus affecting the quality of recommendations. This research proposes clustering and imputation techniques to overcome this problem. The clustering technique used is k-means, while the missing value imputation method uses average values. The imputation results are then implemented in the k-nearest neighbor (KNN) and naïve Bayes algorithms and evaluated based on performance accuracy. Experimental results show an increase in accuracy of 16.48% in the KNN algorithm from 83.52% to 100%. Meanwhile, the naïve Bayes algorithm increased accuracy by 35.30% from 64.70% to 100%.