Ratio Of Missing Data Research Articles

Stream salinity prediction in data-scarce regions: Application of transfer learning and uncertainty quantification

Scarcity of stream salinity data poses a challenge to understanding salinity dynamics and its implications for water supply management in water-scarce salt-prone regions around the world. This paper introduces a framework for generating continuous daily stream salinity estimates using instance-based transfer learning (TL) and assessing the reliability of the synthetic salinity data through uncertainty quantification via prediction intervals (PIs). The framework was developed using two temporally distinct specific conductance (SC) datasets from the Upper Red River Basin (URRB) located in southwestern Oklahoma and Texas Panhandle, United States. The instance-based TL approach was implemented by calibrating Feedforward Neural Networks (FFNNs) on a source SC dataset of around 1200 instantaneous grab samples collected by United States Geological Survey (USGS) from 1959 to 1993. The trained FFNNs were subsequently tested on a target dataset (1998-present) of 220 instantaneous grab samples collected by the Oklahoma Water Resources Board (OWRB). The framework's generalizability was assessed in the data-rich Bird Creek watershed in Oklahoma by manipulating continuous SC data to simulate data-scarce conditions for training the models and using the complete Bird Creek dataset for model evaluation. The Lower Upper Bound Estimation (LUBE) method was used with FFNNs to estimate PIs for uncertainty quantification. Autoregressive SC prediction methods via FFNN were found to be reliable with Nash Sutcliffe Efficiency (NSE) values of 0.65 and 0.45 on in-sample and out-of-sample test data, respectively. The same modeling scenario resulted in an NSE of 0.54 for the Bird Creek data using a similar missing data ratio, whereas a higher ratio of observed data increased the accuracy (NSE = 0.84). The relatively narrow estimated PIs for the North Fork Red River in the URRB indicated satisfactory stream salinity predictions, showing an average width equivalent to 25 % of the observed range and a confidence level of 70 %.

A novel approach for missing data recovery and fault nodes detection in wireless sensor networks

SummaryEnsuring data integrity in wireless sensor networks (WSNs) is crucial for accurate monitoring, yet missing data due to sensor faults present a significant challenge. This research introduces an innovative approach that integrates advanced data recovery techniques with leading‐edge methods to address this issue. The system begins by identifying and isolating fault nodes using a specialized algorithm that analyzes network behavior. By applying fuzzy density‐based spatial clustering of applications with noise (FDBSCAN), potential fault nodes are precisely located based on deviations from expected patterns. Subsequently, an intelligent missing data recovery mechanism powered by bidirectional long short‐term memory (Bi‐LSTM) networks takes action. The Bi‐LSTM model is trained on existing sensor data to capture intricate patterns and dependencies, enabling accurate prediction and reconstruction of missing values caused by identified faults. The synergy between Bi‐LSTM for missing data recovery and FDBSCAN for fault node detection comprehensively addresses the missing data problem in WSNs. In missing data recovery, it demonstrates low mean absolute deviation (MAD) ranging from 0.021 to 0.13 and mean squared deviation (MSD) ranging from 0.0025 to 0.05 across various missing data ratios. Data reliability remains consistently high at 96% to 98%, even with up to 80% missing data. For fault node detection, the approach achieves precision of 95.7%, recall of 96.3%, F1‐score of 96.1%, and accuracy of 97.4%, outperforming existing techniques. The computational cost during training is noted at 5.79 h, presenting a limitation compared to other methods. This research highlights the importance of integrating fault node detection into missing data recovery mechanisms, presenting an innovative solution for the advancement of WSNs.

Bin.INI: An ensemble approach for dynamic data streams

Class imbalance and concept drifts could deteriorate the performance of classifiers in data stream learning as their co-occurrence presents a complicated learning scenario. This situation becomes more complicated if missing values appear in the data. To tackle this challenge, this paper proposes a novel ensemble method, BINary classification in Incomplete, Non-stationary, and Imbalanced data streams (Bin.INI). As part of Bin.INI, Adapt2Regress builds a linear regression models pool using completely available minority subspaces, monitored under a novel sigmoid-decay function to provide imputation support in a drifting data environment. We use Min++ to balance the data in terms of imbalance complexity, a resampling component proposed earlier. Additionally, the 2-STage Ensemble Pruning technique (2STEP) is proposed which groups the classifiers based upon multiple diversity measures at the first step and selects the best subset using a novel custom metric, Sigmoid-Weighted Imbalance Score, at the second step. By using Adapt2Regress, Min++, and 2STEP, the proposed method can efficiently deal with incomplete, imbalanced, and drifting data in data streams. Experiments are conducted on 350 highly imbalanced data streams containing a variety of concept drifts. 300/350 data streams contain different ratios of missing data (5%–30%) in the minority space. Experiments reveal a noticeable decline in the performance of state-of-the-art approaches when faced with 15% or more missing data, while Bin.INI maintains its performance.

The Effects of Missing Data Handling Methods on Reliability Coefficients: A Monte Carlo Simulation Study

This study holds significant implications as it examines the impact of different missing data handling methods on the internal consistency coefficients. With Monte Carlo simulations, we manipulated number of items, true reliability, sample size, missing data ratio and mechanisms, to compare relative bias of reliability coefficients. The reliability coefficients under scrutiny in this study encompass Cronbach's Alpha, Heise & Bohrnsted's Omega, Hancock & Mueller's H, Gölbaşı-Şimşek & Noyan's Theta G, Armor's Theta, and Gilmer-Feldt coefficients. Our arsenal of techniques includes single imputation methods like zero, mean, median, and regression imputation, as well as multiple imputation approaches like expectation maximization and random forest. We also employ the classic deletion method known as listwise deletion. The findings suggest that, for missing completely at random (MCAR) or missing at random (MAR) data, single imputation approaches (excluding zero imputation) may still be preferable to expectation maximization and random forest imputation, thereby underscoring the importance of our research.

“Sparse + Low-Rank” tensor completion approach for recovering images and videos

Recovering color images and videos from highly undersampled data is a fundamental and challenging task in face recognition and computer vision. By the multi-dimensional nature of color images and videos, in this paper, we propose a novel tensor completion approach, which is able to efficiently explore the sparsity of tensor data under the discrete cosine transform (DCT). Specifically, we introduce two “sparse + low-rank” tensor completion models as well as two implementable algorithms for finding their solutions. The first one is a DCT-based sparse plus weighted nuclear norm induced low-rank minimization model. The second one is a DCT-based sparse plus p-shrinking mapping induced low-rank optimization model. Moreover, we accordingly propose two implementable augmented Lagrangian-based algorithms for solving the underlying optimization models. A series of numerical experiments including color image inpainting and video data recovery demonstrate that our proposed approach performs better than many existing state-of-the-art tensor completion methods, especially for the case when the ratio of missing data is high.

Improved Hybrid K-Nearest Neighbours Techniques in Segmentation of Low-Grade Tumor and Cerebrospinal Fluid

In image segmentation, identifying information or object detection in medical im-ages is crucial, particularly information that is harder to spot in magnetic resonance imaging (MRI) of low-grade tumors or cerebrospinal fluid (CSF). To address the aforementioned problems associated with missing data in MRI images and the low quality of MRI images that required longer processing times, this research is to seg-ment brain tumors or detect CSF in four-dimensional MRI images. A new hybrid k-nearest neighbors (k-NN) framework is also proposed, which consists of three tech-niques: correlation matrices of discrete Fourier transform (CM-DFT), Laplace Eigen maps of locally preserving projection (LELPP), and a hybrid GrabCut hidden Markov model of k-mean clustering (GCHMkC). The combination of the Hidden Markov Model (HMM) and the k-mean clustering technique is known as the Hidden Markov Model of the k-mean clustering method (HMMkC). To begin with, the Graph Cut and Support Vector Machine (GCSVM) and the GCHMkC approach are combined. The method increased the quality of the images suggested by the methodology, achieving an accuracy of 99.83%, a sensitivity of 99.99%, a specificity of 99.8%, and a computa-tional execution time of 14.9 seconds. Second, a technique called CM-DFT is sug-gested to improve MRI images while resolving the issue of missing imputation data. The accuracy of the MRI image datasets was improved to 99.84%, the time-lag in the hybrid k-NN algorithm was reduced to 99%, the missing data ratio was reduced to 0.9%, 10%, and 12%, and the correctness of the imputed data was improved to 1.533 seconds with computational execution. Thirdly, the nonlinear data is reduced and unnecessary features are eliminated using the Laplace Eigen maps of locally pre-serving projection (LELPP) approach. The hybrid k-NN algorithm used by the tech-nique yields results with 99% accuracy and an execution duration of 2.42 seconds.

Not missing at random: Missing data are associated with clinical status and trajectories in an electronic monitoring longitudinal study of bipolar disorder

There is limited information on the association between participants' clinical status or trajectories and missing data in electronic monitoring studies of bipolar disorder (BD). We collected self-ratings scales and sensor data in 145 adults with BD. Using a new metric, Missing Data Ratio (MDR), we assessed missing self-rating data and sensor data monitoring activity and sleep. Missing data were lowest for participants in the midst of a depressive episode, intermediate for participants with subsyndromal symptoms, and highest for participants who were euthymic. Over a mean ± SD follow-up of 246 ± 181 days, missing data remained unchanged for participants whose clinical status did not change throughout the study (i.e., those who entered the study in a depressive episode and did not improve, or those who entered the study euthymic and remained euthymic). Conversely, when participants' clinical status changed during the study (e.g., those who entered the study euthymic and experienced the occurrence of a depressive episode), missing data for self-rating scales increased, but not for sensor data. Overall missing data were associated with participants’ clinical status and its changes, suggesting that these are not missing at random.

Optimizing Imputation for Educational Data: Exploring Training Partition and Missing Data Ratios

The integration of machine learning in educational data analysis presents challenges regarding the availability of sufficient training data, especially in the context of high missing data ratios. These challenges arise from data partitioning practices, resulting in smaller datasets and less precise models. Behavioral scientists have increasingly incorporated machine learning into propensity score estimation, necessitating investigations into the most effective training and testing partitioning methods for machine learning-based imputation. To address this gap in the literature, our Monte Carlo experiment examines the impact of partitioning methods and missing data ratios. Simulated datasets, featuring missing ratios of 10%, 30%, 50%, and 70%, are divided into training and testing sets, ranging from 80–20 to 20–80. Results indicate that each imputation method delivers highly accurate average treatment effects. However, in the context of maintaining covariate balance across diverse conditions, complex ensemble methods outperform artificial neural networks. A real-data comparison (Study II) further underscores that the adoption of sophisticated machine learning techniques significantly enhances covariate balance. This research contributes valuable insights into the development of machine learning-based imputation methods, with a specific focus on scenarios characterized by high missing data ratios, in educational data analysis.

Characteristics of Missing Data in Single-Case Experimental Designs: An Investigation of Published Data.

Single-case experimental designs (SCEDs) have grown in popularity in the fields such as education, psychology, medicine, and rehabilitation. Although SCEDs are valid experimental designs for determining evidence-based practices, they encounter some challenges in analyses of data. One of these challenges, missing data, is likely to be occurred frequently in SCEDs research due to repeated measurements over time. Since missing data is a critical factor that can weaken the validity and generalizability of a study, it is important to determine the characteristics of missing data in SCEDs, which are especially conducted with a small number of participants. In this regard, this study aimed to describe missing data features in SCEDs studies in detail. To accomplish this goal, 465 published SCEDs studies within the recent 5 years in six journals were included in the investigation. The overall results showed that the prevalence of missing data among SCEDs articles in at least one phase, as at least one data point, was approximately 30%. In addition, the results indicated that the missing data rates were above 10% within most studies where missing data occurred. Although missing data is so common in SCEDs research, only a handful of studies (5%) have handled missing data; however, their methods are traditional. In analyzing SCEDs data, several methods are proposed considering missing data ratios in the literature. Therefore, missing data rates determined in this study results can shed light on the analyses of SCEDs data with proper methods by improving the validity and generalizability of study results.

Reconstruction of flow field with missing experimental data of a circular cylinder via machine learning algorithm

In this study, artificial neural networks (ANNs) have been implemented to recover missing data from the particle image velocimetry (PIV), providing quantitative measurements of velocity fields. Due to laser reflection or lower intensity of particles in the interrogation area, the reconstruction of erroneous velocity vectors is required. Therefore, the distribution of time-averaged and normalized flow characteristics around a circular cylinder has been demonstrated as streamwise and cross-stream velocities at Re = 8000. These velocity components have been given for different regions at x/D = 0.5, x/D = 1.25, x/D = 2, and y/D = 0. These stations have been chosen to estimate missing data for near-wake, mid-wake, far-wake, and symmetry regions. The missing data ratios (A*) for 0.5 ≤ x/D ≤ 2 are A* = 3.5%, 7%, and 10%. In addition, these values are A* = 4%, 8%, and 12% for y/D = 0, while A* = 7.5% for the shaded region. The increment of area positively affects the estimation results for near-wake and mid-wake regions. Moreover, the errors tend to decrease by moving away from the body. At y/D = 0, increasing the area negatively influences the prediction of the results. The mean velocity profiles of predicted and experimental data have also been compared. The missing data have been predicted with a maximum percentage error of 3.63% for horizontal stations. As a result, the ANN model has been recommended to reconstruct PIV data.

AGNP: Network-wide short-term probabilistic traffic speed prediction and imputation

The data-driven Intelligent Transportation System (ITS) provides great support to travel decisions and system management but inevitably encounters the issue of data missing in monitoring systems. Hence, network-wide traffic state prediction and imputation is critical to recognizing the system level state of a transportation network. Abundant research works have adopted various approaches for traffic prediction and imputation. However, previous methods ignore the reliability analysis of the predicted/imputed traffic information. Thus, this study originally proposes an attentive graph neural process (AGNP) method for network-level short-term traffic speed prediction and imputation, simultaneously considering reliability. Firstly, the Gaussian process (GP) is used to model the observed traffic speed state. Such a stochastic process is further learned by the proposed AGNP method, which is utilized for inferring the congestion state on the remaining unobserved road segments. Data from a transportation network in Anhui Province, China, is used to conduct three experiments with increasing missing data ratio for model testing. Based on comparisons against other machine learning models, the results show that the proposed AGNP model can impute traffic networks and predict traffic speed with high-level performance. With the probabilistic confidence provided by the AGNP, reliability analysis is conducted both numerically and visually to show that the predicted distributions are beneficial to guide traffic control strategies and travel plans.

Defining longitudinal trajectory of body mass index percentile and predicting childhood obesity: methodologies and findings in the Boston Birth Cohort.

Overweight or obesity (OWO) in school-age childhood tends to persist into adulthood. This study aims to address a critical need for early identification of children at high risk of developing OWO by defining and analyzing longitudinal trajectories of body mass index percentile (BMIPCT) during early developmental windows. We included 3029 children from the Boston Birth Cohort (BBC) with repeated BMI measurements from birth to age 18 years. We applied locally weighted scatterplot smoothing with a time-limit scheme and predefined rules for imputation of missing data. We then used time-series K-means cluster analysis and latent class growth analysis to define longitudinal trajectories of BMIPCT from infancy up to age 18 years. Then, we investigated early life determinants of the BMI trajectories. Finally, we compared whether using early BMIPCT trajectories performs better than BMIPCT at a given age for predicting future risk of OWO. After imputation, the percentage of missing data ratio decreased from 36.0% to 10.1%. We identified four BMIPCT longitudinal trajectories: early onset OWO; late onset OWO; normal stable; and low stable. Maternal OWO, smoking, and preterm birth were identified as important determinants of the two OWO trajectories. Our predictive models showed that BMIPCT trajectories in early childhood (birth to age 1 or 2 years) were more predictive of childhood OWO (age 5-10 years) than a single BMIPCT at age 1 or 2 years. Using longitudinal BMIPCT data from birth to age 18 years, this study identified distinct BMIPCT trajectories, examined early life determinants of these trajectories, and demonstrated their advantages in predicting childhood risk of OWO over BMIPCT at a single time point.

A hybrid approach of missing data imputation for upper gastrointestinal diagnosis

Gastrointestinal and liver diseases (GILDs) are the major causes of death and disability in Middle East. The investigation of upper gastrointestinal (GI) symptoms of a medically limited area resource is a challenge. The analysis of real-world clinical data of upper gastrointestinal (GI) using data mining techniques often is facing observations that the data contains missing values. In this paper, the proposed approach to missing data imputation is accomplished for categorical data onto upper GI diseases to apply the feature selection and classification algorithms with accurate and effective results for diagnosing upper GI diseases. This approach is evaluated by implementing experimental framework to apply five phases. These phases aim at partitioning the dataset to eight different datasets; with various ratio of missing data, performing the feature selection, imputing the missing data, classifying the imputed data, and finally, evaluating the outcome using k-fold cross validation for nine evaluation measures.

KAYIP VERİ ATAMA YÖNTEMLERİNİN GENELLENEBİLİRLİK KURAMI G VE PHİ KATSAYILARINA ETKİSİ

Generalizability theory is a theory enabling comprehensive reliability analyses by dealing with several variability source like item, task, rater, time, etc. However, missing data causes an inability in making analyses and predictions. In literature, there is limited number of studies comparing missing data imputation methods in generalizability theory. In this direction, the aim of the study is to compare G and Phi coefficients obtained from generalizability theory analyses according to full data matrix and missing data imputation methods with different sample sizes and missing data rates. In this research, a Monte Carlo simulation study was conducted in order to compare performances of missing data imputation methods for missing data for two-facet crossed designs differently from limited studies in literature. In the simulation study, 36 conditions as “number of examine x missing data ratio x method (3 x 4 x 3)” were investigated. In addition, G and Phi coefficients were calculated for full data matrices for each sample size. Bias between G and Phi coefficients was compared by using imputation methods with these calculated values. As a result of this comparison, it is found that reliability predictions closest to the full data come from EM method instead of missing data. If EM is used instead of missing data, reliability predictions quite similar to reliability coefficients predicted from full data matrix is found in most of the conditions. The lowest level of relative error and bias for reliability coefficients is found with EM and LR methods respectively. PM is the method with the highest error and bias considering all conditions. In addition, the current study found that as the missing data rate increase, difference, error and bias values increase for all methods according to full data matrix.

Semi-GAN: An Improved GAN-Based Missing Data Imputation Method for the Semiconductor Industry

Complete data are required for the operation, maintenance, and detection of faults in semiconductor equipment. Missing data occur frequently because of defects such as sensor, data storage, and communication faults, leading to reductions in yield, quality, and productivity. Although many attempts have been made to solve this problem in other fields, few studies have specifically addressed data imputation in the semiconductor industry. In this study, an improved generative adversarial network (GAN)-based missing data imputation for the semiconductor industry called Semi-GAN is proposed. This study introduces a machine learning approach for dealing with data imputation in the semiconductor industry. The proposed method was applied to real data and evaluated using traditional techniques. In particular, the proposed method showed excellent results compared to traditional attribution methods when all missing data ratios in the experiments were less than 20%. It was also observed to be superior when simple and repetitive patterns were omitted rather than repetitive but not simple patterns.

A machine learning approach for spatiotemporal imputation of MODIS chlorophyll-a

ABSTRACT The Moderate Resolution Imaging Spectroradiometer (MODIS) level-3 chlorophyll-a (Chl-a) product is one of the widely used ocean colour products that is often used for water quality monitoring of marine ecosystems. However, this product includes a large amount of missing data due to high surface reflectance and cloudy conditions that inevitably affect its suitability for spatiotemporal analysis of water quality. The objective of this study was to compare four Machine Learning (ML) techniques including K-nearest neighbour (KNN), Support Vector Regression (SVR), Random Forest Regression (RFR), and Artificial Neural Network (ANN) with well-known Data Interpolation Empirical Orthogonal Function (DINEOF) method for spatiotemporal missing imputation of MODIS Chl-a. The Southern Caspian Sea, which has a high Chl-a concentration, was selected as the case study. A cross-validation approach ranging missing data ratio from 0.1 to 0.8 was implemented to investigate the optimal parameters of the models and compare their performance for missing imputation. The results indicated that all ML models, except KNN, outperformed the DINEOF method for missing imputation of MODIS Chl-a. The SVR with the highest accuracy and the lowest variability of errors had the best performance among the five competing models, while the KNN showed the worst performance. The main reason for the better accuracy of the SVR than the other models is its structural risk minimization procedure that leads to the better generalization of the SVR model. The current results showed that the ML techniques used in the current study, the SVR in particular, are able to produce reliable imputations of the MODIS Chl-a missing data and can be a useful tool in water quality monitoring of marine ecosystems.

Python-based preprocessing for applying machine learning in breast cancer metastasis prediction.

e13558 Background: Artificial intelligence (AI) and machine learning (ML) have outstanding contributions in oncology. One of the applications is the early detection of breast cancer. Recently, several ML and data mining techniques have been used for both detection and classification of breast cancer cases. It is found that about 25% of breast cancer cases have an aggressive cancer at diagnosis time, with metastatic spread. The absence or presence of metastatic spread largely determines the patient’s survival. Hence, early detection is very important for reducing cancer mortality rates Methods: This study aims at applying ML and data mining, using AI techniques, for exploring and preprocessing breast cancer dataset, before building the ML classification Model for breast cancer metastasis prediction. The model will be implemented for mass screening, to prioritize patients who are more likely to develop metastases. A dataset of breast cancer cases was provided by the Oncology and Nuclear Medicine Department, Faculty of Medicine, Alexandria University. It contains clinical records of 5236 patients, diagnosed with breast cancer. ML libraries in Python programming language was used to explore the dataset and determine ratio of missing data, define data types, redundant data, and specify class label and predictors that to be used for the classification model. Results: In this work, the results showed that missing data ratio in some columns exceeds 90%, there are redundant features to be eliminated, data type conversion and feature reduction should be applied to prepare the data. Conclusions: Based on the previous findings, it is recommended to use ML preprocessing python libraries to prepare the dataset before building ML classification model of breast cancer metastasis prediction.

Missing Value Imputation in Stature Estimation by Learning Algorithms Using Anthropometric Data: A Comparative Study

Estimating stature is essential in the process of personal identification. Because it is difficult to find human remains intact at crime scenes and disaster sites, for instance, methods are needed for estimating stature based on different body parts. For instance, the upper and lower limbs may vary depending on ancestry and sex, and it is of great importance to design adequate methodology for incorporating these in estimating stature. In addition, it is necessary to use machine learning rather than simple linear regression to improve the accuracy of stature estimation. In this study, the accuracy of statures estimated based on anthropometric data was compared using three imputation methods. In addition, by comparing the accuracy among linear and nonlinear classification methods, the best method was derived for estimating stature based on anthropometric data. For both sexes, multiple imputation was superior when the missing data ratio was low, and mean imputation performed well when the ratio was high. The support vector machine recorded the highest accuracy in all ratios of missing data. The findings of this study showed appropriate imputation methods for estimating stature with missing anthropometric data. In particular, the machine learning algorithms can be effectively used for estimating stature in humans.

Parameter estimation in spatial econometric models with non-random missing data

ABSTRACT This study examines the problem of parameter estimation in spatial econometric/social interaction models with non-random missing outcome data. First, we construct a sample selection model considering spatial lag (autoregressive) dependence. Then, we suggest a parameter estimation method for this model by slightly modifying the Bayesian Markov chain Monte Carlo algorithm proposed in an existing study. A simple illustration indicates that the proposed parameter estimation method performs well overall if the spatial autocorrelation is moderate (spatial parameter equals 0.5 or less), even under a relatively high missing data ratio (around 40%).

Traffic Estimation for Large Urban Road Network with High Missing Data Ratio.

Intelligent transportation systems require the knowledge of current and forecasted traffic states for effective control of road networks. The actual traffic state has to be estimated as the existing sensors does not capture the needed state. Sensor measurements often contain missing or incomplete data as a result of communication issues, faulty sensors or cost leading to incomplete monitoring of the entire road network. This missing data poses challenges to traffic estimation approaches. In this work, a robust spatio-temporal traffic imputation approach capable of withstanding high missing data rate is presented. A particle based approach with Kriging interpolation is proposed. The performance of the particle based Kriging interpolation for different missing data ratios was investigated for a large road network comprising 1000 segments. Results indicate that the effect of missing data in a large road network can be mitigated by the Kriging interpolation within the particle filter framework.