Mean Imputation Method Research Articles

Improving Air Quality Data Reliability through Bi-Directional Univariate Imputation with the Random Forest Algorithm

Forecasting the future levels of air pollution provides valuable information that holds importance for the general public, vulnerable populations, and policymakers. High-quality data are essential for precise and reliable forecasts and investigations of air pollution. Missing observations arise when the sensors utilized for assessing air quality parameters experience malfunctions, which result in erroneous measurements or gaps in the dataset and hinder the data quality. This research paper presents a novel approach for imputing missing values in air quality data in a univariate approach. The algorithm employs the random forest (RF) algorithm to impute missing observations in a bi-directional (forward and reverse in time) manner for air quality (particulate matter less than 2.5 μm (PM2.5)) data from the Republic of Serbia. The algorithm was evaluated against simple methods, such as the mean and median imputation methods, for missing observations over durations of 24, 48, and 72 h. The results indicate that our algorithm yielded comparable error rates to the median imputation method for all periods when imputing the PM2.5 data. Ultimately, the algorithm’s higher computational complexity proved itself as not justified considering the minimal error decrease it achieved compared with the simpler methods. However, for future improvement, additional research is needed, such as utilizing low-code machine learning libraries and time-series forecasting techniques.

Ranked set sampling imputation methods in presence of correlated measurement errors

Very little attention has been paid by the authors for providing improved imputation methods under correlated measurement errors (CMEs) using various sampling designs. This article addresses several power ratio imputation methods and the resulting estimators under ranked set sampling (RSS) when CMEs are present. The mean square error (MSE) is developed to assess how well the suggested estimators work under CMEs. The effectiveness of the proposed imputation methods and corresponding resultant estimators is assessed by a comprehensive simulation utilizing an artificially generated population. Further, an application of the proposed imputation methods is also provided using real data from Sweden’s municipalities consisting of the total number of seats in 1982 in municipal council taken as study variable and the number of conservative seats in 1982 in municipal council taken as auxiliary variable. The CMEs might arise due to common surveyor biases if the same surveyor collects information on both the “total number of seats in municipal council” and “the number of conservative seats in that municipal council”. The simulation and real data results show that the proposed power ratio imputation methods outperform the mean, conventional ratio, and product imputation methods.

Synthetic imputation methods for domain mean under simple random sampling

The estimation of domain mean in presence of missing data is a significant issue in sample surveys. Literature contains no imputation method to estimate the domain mean in the presence of missing data using bivariate auxiliary information. The present study uses simple random sampling to propose several novel synthetic domain mean imputation methods and the corresponding resulting estimators based on bivariate auxiliary information in the case of missing data. To evaluate how well the suggested novel imputation methods and the resulting estimators perform, the mathematical equation of mean square errors is derived. A thorough simulation experiment is also carried out using a population drawn artificially from a normal distribution. Furthermore, an application is also offered based on real data.

Comparative Analysis of Rainfall Data Sources in Agro-Ecological Zones of Sri Lanka: Expensive Meteorological Measurements vs Freely Available NASA Data

Rainfall information contributes significantly to weather-related research as it can be directly applied to many theories and models such as climate change, disaster management, environmental assessments. Therefore, the availability of long-term meteorological data and their accuracy are crucial. At the same time the access to these data from different sources have their own drawbacks. For example, meteorological department of Sri Lanka is responsible of collecting rainfall data. However, their data are very expensive, and their data are not available for all desired locations. On the other hand, the freely available satellite data sources have a question of reliability of the data although no missing data are found. Thus, a comparative assessment of rainfall data was carried out from two distinct sources: meteorological department of Sri Lanka and freely available National Aeronautics and Space Administration (NASA) satellite-based data. The objective of this study was to compare and discern the distinctions in rainfall data obtained from these sources for the benefit of future usage of the data in various studies. The study was carried out in different agro-ecological Zones (wet, dry, and intermediate) of Sri Lanka for the period of 1981-2021. The stations were selected randomly to represent the zones. Identified missing values in the meteorological dataset were subsequently addressed using the mean imputation method. Statistical procedures were conducted on the non-normal rainfall data to identify similarities between the sources within individual agro-ecological zones. Results revealed that there is a statistically significant difference at 5% between the sources (NASA and meteorological measurements) in wet and intermediate zones. However, the comparison of the dry zone resulted in inconclusive decisions both at 5% and 1% significance levels. In conclusion, it is statistically advisable to use meteorological rainfall measurements over NASA data for the wet and intermediate zones.
 
 Keywords: Rainfall, Agro-ecological zones, Meteorological data, NASA data, Non-normal data

Predictive Analysis in Nephrology: Evaluating Machine Learning Models for Chronic Kidney Disease Prediction

Abstract: In the realm of healthcare, early detection and accurate diagnosis are critical for effective treatment, especially for diseases that are often undiagnosed until later stages. This research presents an Automated Classification System focused on Chronic Kidney Disease (CKD), a prevalent condition frequently undetected in its early stages. Leveraging advanced machine learning techniques, the study aims to overcome the traditional reliance on doctors' intuition and experience by providing a data-driven approach for CKD diagnosis. Utilizing a comprehensive dataset, the research explores various data pre-processing methods, including innovative approaches for handling missing values and data integrity, which surpass the conventional mean and mode-based imputation methods prevalent in existing literature. Through exploratory data analysis and the application of diverse machine learning algorithms, the study seeks to identify the most effective model for predicting CKD risk. The findings of this research have significant implications for enhancing clinical decisions, offering a path towards more accessible, efficient, and accurate diagnosis of Chronic Kidney Disease, ultimately contributing to improved patient outcomes and healthcare practices

Evaluation of Imputation Methods for Handling Missing Data in Mechanical Materials Datasets

Mechanical design materials are integral to advancing modern technologies due to their diverse mechanical properties. These properties are crucial in determining the material’s suitability for various engineering applications. However, research on mechanical materials often encounters missing data, which can lead to biased results and reduced statistical power. While several imputation methods exist to handle missing data, there is a lack of focused studies evaluating their performance in the context of mechanical materials. To address this gap, a comprehensive dataset was obtained, and 10% of the original data for ultimate tensile strength (Su) and yield strength (Sy) were intentionally deleted. Four imputation methods—mean imputation, random fill, regression imputation, and k-nearest neighbors (KNN) imputation—were employed to restore the missing data. The performance of these methods was evaluated using Pearson’s correlation, multiple linear regression, and permutation feature importance. The results showed that mean and KNN imputation methods provided the closest match to the original data, while regression imputation also performed well with minor deviations. Random fill was the least reliable method. These findings provide guidance on selecting appropriate imputation techniques for mechanical materials datasets, ultimately improving the robustness of future research.

Identifying Critical Success Factors of an Emergency Information Response System Based on the Similar-DEMATEL Method

An emergency information response system (EIRS) is a system that utilizes various intelligence technologies to effectively handle various emergencies and provide decision support for decision-makers. As critical success factors (CSFs) in EIRS play a vital role in emergency management, it is necessary to study the CSFs of EIRS. Most previous studies applied the Decision Experiment and Decision-Making Trial and Evaluation Laboratory (DEMATEL) method with complete evaluation information to identify CSFs. Due to the complexity of the decision-making environment when identifying CSFs of EIRS, decision-makers sometimes cannot provide complete evaluation information during the decision-making process. To fill this gap, this paper provided a Similar-DEMATEL method to impute the missing values and identify CSFs of EIRS, which may avoid the dilemma of decision distortion and make decision-making results more accurate. It is found that the factors of Information mining capability, Equipment support capability, Monitoring and early warning capability, and Organization participation capability are the CSFs in EIRS. Our proposed method differs from previous research, such as the mean imputation method, to impute the missing values. We compared the differences between the proposed method and the mean imputation method and gave the advantages of the proposed method. Our method focuses more on uncertain decision-making environments, which is conducive to improving the efficiency of EIRS in emergency management, and therefore it is more widely adopted.

Novel Logarithmic Imputation Methods Under Ranked Set Sampling

ABSTRACT The data we encounter in real life often contain missing values. In sampling methods, missing value imputation is done with different methods. This article proposes novel logarithmic type imputation methods for estimating the population mean in the presence of missing data under ranked set sampling (RSS). According to the determined theoretical results, the proposed imputation methods are found to be the most efficient in comparison to popularly known imputation methods like mean imputation, Al-Omari and Bouza (2014) imputation methods, Sohail et al. (2018) imputation methods, and Bhushan and Pandey (2016) type imputation methods utilizing RSS. Apart from this, a simulation study has been accomplished utilizing artificially drawn symmetric and asymmetric populations. The outcomes are encountered to be rather satisfactory, showing improvement over all existing imputation methods.

On the efficiency of modified regression-type mean imputation scheme under two-phase sampling

Human-based surveys such as medical and social science surveys are often characterized by non-response or missing observations. In this study, a new class of regression-type mean imputation method that uses X̄n as an estimate of X̄ was suggested. Using partial derivative approach, the MSEs of the class of estimators presented were derived up to first order approximation under two cases. Case I: when the secondary sample S2 of size n(n<n1) is a subset of preliminary sample S1[S2 ⸦ S1] , and Case II: is when secondary sample S2 is a subset of universal set ΩN . Conditions for which the new estimator was more efficient than the other estimators studied were derived. The results of numerical examples through simulations revealed that the suggested class of estimators is more efficient.

A novel data gaps filling method for solar PV output forecasting

This study proposes a modified gaps filling method, expanding the column mean imputation method and evaluated using randomly generated missing values comprising 5%, 10%, 15%, and 20% of the original data on power output. The XGBoost algorithm was implemented as a forecasting model using the original and processed datasets and two sources of solar radiation data, namely, Shortwave Radiation (SWR) from Advanced Himawari Imager 8 (AHI-8) and Surface Solar Radiation Downward (SSRD) from ERA5 global reanalysis data. The accuracy of the two sets of forecasted power output was evaluated using Root Mean Square Error (RMSE) and Mean Absolute Error (MAE). Results show that by applying the proposed gap filling method and using SWR in forecasting solar photovoltaic (PV) output, the improvement in the RMSE and MAE values range from 12.52% to 24.30% and from 21.10% to 31.31%, respectively. Meanwhile, using SSRD, the improvement in the RMSE values range from 14.01% to 28.54% and MAE values from 22.39% to 35.53%. To further evaluate the accuracy of the proposed gap-filling method, the proposed method could be validated using different datasets and other forecasting methods. Future studies could also consider applying the said method to datasets with data gaps higher than 20%.

Multinomial Logistic Factor Regression for Multi-source Functional Block-wise Missing Data.

Multi-source functional block-wise missing data arise more commonly in medical care recently with the rapid development of big data and medical technology, hence there is an urgent need to develop efficient dimension reduction to extract important information for classification under such data. However, most existing methods for classification problems consider high-dimensional data as covariates. In the paper, we propose a novel multinomial imputed-factor Logistic regression model with multi-source functional block-wise missing data as covariates. Our main contribution is to establishing two multinomial factor regression models by using the imputed multi-source functional principal component scores and imputed canonical scores as covariates, respectively, where the missing factors are imputed by both the conditional mean imputation and the multiple block-wise imputation approaches. Specifically, the univariate FPCA is carried out for the observable data of each data source firstly to obtain the univariate principal component scores and the eigenfunctions. Then, the block-wise missing univariate principal component scores instead of the block-wise missing functional data are imputed by the conditional mean imputation method and the multiple block-wise imputation method, respectively. After that, based on the imputed univariate factors, the multi-source principal component scores are constructed by using the relationship between the multi-source principal component scores and the univariate principal component scores; and at the same time, the canonical scores are obtained by the multiple-set canonial correlation analysis. Finally, the multinomial imputed-factor Logistic regression model is established with the multi-source principal component scores or the canonical scores as factors. Numerical simulations and real data analysis on ADNI data show the proposed method works well.

Local mean imputation for handling missing value to provide more accurate facies classification

The search for oil and gas reserves is an ongoing challenge. The possibility of oil and natural gas reserves running out in Indonesia in less than 25 years must also be considered by the industry. Unfortunately, the process of exploration and development of hydrocarbon fields still has its challenges, one of which is knowing the characteristics of hydrocarbon reservoirs. One of these characteristics that should be analyzed is rock facies. This could be done by using core data, well logs, analog studies, and others. However, this method is very expensive and time-consuming. Currently, the machine learning approach is the most popular solution for the classification of rock facies. However, a common issue with this method is the missing values due to recording errors in the field. Meanwhile, a machine learning system requires accurate data to form a good model. Therefore, we propose the use of a simple imputation method to solve the local mean imputation method by considering the uniqueness of each class before performing the imputation process. We also implemented several popular boosting methods in machine learning such as comparing Extreme Gradient Boosting and Light Gradient Boosting with several machine learning algorithms that are often used in solving facies classification problems such as Random Forest, Decision Tree, Support Vector Machine, and K-Nearest Neighbor. According to our results, the Local Mean imputation method improved the model's performance, where the perceived increase was between 1-10% and the model with the best performance was generated by the Random Forest method.

Salt Commodity Data Clustering Using Fuzzy C-Means

Indonesia has a sea area of about 96,079.15 km2 rich in natural resources. The advantage is in the form of abundant natural resources in the sea including fish and salt. Several regions such as Cirebon, Indramayu, Rembang, and Madura contribute to the salt pond commodity which is extremely valuable for Indonesia. These resources need to be monitored and inventoried appropriately. Local people are difficult and increasingly cornered to compete in the salt trade. For the strategic commodity group, by maintaining the stability of the salt commodity in the community with its irreplaceable function, the salt trade system is regulated. The purpose of this research is to use the Fuzzy C-Means Clustering method for grouping national salt commodities. This study used a test with 10 clusters for the distribution of training and testing data purposes. The research trials were carried out using the mean imputation method and grouping using the Fuzzy C-Means Clustering method. The test results of the Fuzzy C-Means Clustering method using the Silhouette Coefficient method show that the Fuzzy C-Means Clustering method with the calculation of the closest distance using Manhattan Distance is the best choice in research with the results obtained in the Silhouette Coefficient assessment is at a value of 0.274880 with a value of k = 2.

Optimizing asphalt mix design through predicting the rut depth of asphalt pavement using machine learning

Generally, when asphalt concrete (AC) is in the design phase, the rutting development of the actual pavement is always not considered. Traditional simulative wheel-tracking tests, which are used to evaluate the rutting resistance of the designed AC, are difficult to accurately predict the rut depth of actual pavement in the realistic climate condition and traffic. To develop an asphalt mix design approach and avoid early serious pavement damage (rutting), the data relating to rut depth of AC pavements were extracted from the LTPP program,. 27 input features about climate condition, traffic, pavement structure, and pavement materials properties were selected based on collinearity diagnostics and impurity-base feature importance methods. This study also analyzed the effect of different imputation methods used to handle missing variables of the dataset on the performance of ML models. Four different ML algorithms including Support Vector Regression (SVR), Random Forest, Artificial neural networks (ANN), Gradient boosting, and multiple linear regression were trained to predict rut depth. The performances of these models were evaluated by calculating different performance measurement scoring metrics, such as coefficient of determination (R2), root means square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), and scatter index (SI). Finally, an asphalt mix design procedure based on the best ML model was proposed. The results showed that compared to other imputation methods, the mean imputation method can result in the best performance of ML models. Compared with other ML models, Gradient boosting was the optimum model to predict rut depth for the imputation dataset (R2 = 0.87) and full dataset (R2 = 0.92). Additionally, each ML model except for Random Forest can achieve better accuracy than that of the calibrated rutting prediction models of HDM-4 and MEPDG from the literature. The proposed mix design procedure was implemented to design AC for the surface layer of a pavement project. The rut depth of the pavement, whose surface AC was the same as the recommended mixture design by the proposed method, is lower than the pavement failure criterion (MEPDG) after 20 years. It implicated that the proposed procedure based on the ML model can effectively avoid early intolerant rutting damage of flexible pavement.

Semi parametric Estimators for Quantile Model via LASSO and SCAD with Missing Data

In this study, we made a comparison between LASSO & SCAD methods, which are two special methods for dealing with models in partial quantile regression. (Nadaraya & Watson Kernel) was used to estimate the non-parametric part ;in addition, the rule of thumb method was used to estimate the smoothing bandwidth (h). Penalty methods proved to be efficient in estimating the regression coefficients, but the SCAD method according to the mean squared error criterion (MSE) was the best after estimating the missing data using the mean imputation method

Lifetime evaluation of mechanical gyroscope based on incomplete data and censored data

The statistical usage data of the mechanical gyroscope is incomplete and censored, and how to evaluate the lifetime reasonably according to the data is of great significance in this context. In this paper, the incomplete data is sorted and filled based on the field use, which draws on the idea of mean imputation method and regression analysis method. And the nonparametric statistical method in the survival analysis is applied to evaluate reliability and lifetime. The results are in good agreement with the actual use, indicating that the application of the above method can be used quickly and effectively to valuate reliability and lifetime under the condition of incomplete and censored data, which provides a basis for formulating the time to first overhaul and lifetime index. At the same time, the paper points out the possible reliability problems and improvement direction according to statistics and evaluation results.

0159 Sleep-wake stability and variability in the middle-aged adult population: a UK Biobank study

Abstract Introduction Given the increasing use of consumer, wrist-worn devices with triaxial accelerometry (actigraphy), understanding whether 24-hr activity patterns are associated with specific mental and physical health deficits is of paramount importance. The UK Biobank, a community-based sample of adults in the United Kingdom, provides an opportunity to examine this question given its size (more than 100,000 actigraphy records) and scope (association health records). As a first step in understanding these relationships, understanding the impact of missing data, ways to interpolate missing data, and the general characteristics of 24-hr patterns is necessary. Methods A subset (n=70) of complete (i.e., no missing data) actigraphy records from individuals participating in the UK Biobank were used to examine the impact of missing data on intradaily variability (IV; rhythm fragmentation) and interdaily stability (IS; rhythm regularity). Data were intentionally removed and imputed and the impact of the manipulations was examined with Bland-Altman statistics. After determining the best imputation method and the limits of the method, it was applied to missing data in the full cohort, resulting in 82,840 actigraphy records. These were examined for IV and IS, as well as relative amplitude and the timing of L5 and M10. Data were categorized by age, gender, ethnicity, BMI and material deprivation. ANOVA with eta² effect sizes were used for comparisons. Data shown as median (interquartile range). Results Relative to IV and IS, median and mean imputation methods corrected for missing data of up to 24 hours. Linear imputation of long (>3 hours) missing data worsened IV and IS scores. Non-parametric patterns in the general population were within expected range, with IV=0.89(0.74-1.05), IS=0.55(0.46-0.63), RA=0.90(0.85-0.99), L5-timing=00:52 (00:07-01:37), and M10-timing=08:25(07:40-09:20). Stratifying by gender, age, ethnicity, BMI and material deprivation resulted in significant differences within all groups (p<0.0001), but small effect sizes (0.0019-0.00052). Conclusion Median imputation is useful for correcting for missing data (<24 hours) when examining IV and IS. Non-parametric analysis of the general population indicated values within the expected range. Even though there were significant differences in non-parametric outcomes between genders, age groups, ethnicities, body-mass index and material deprivation, none of these had significant effect sizes. Support (If Any) This research has been conducted using the UK Biobank Resource

Multi-Objective Feature Selection With Missing Data in Classification

Feature selection (FS) is an important research topic in machine learning. Usually, FS is modelled as a bi-objective optimization problem whose objectives are: 1) classification accuracy; 2) number of features. One of the main issues in real-world applications is missing data. Databases with missing data are likely to be unreliable. Thus, FS performed on a data set missing some data is also unreliable. In order to directly control this issue plaguing the field, we propose in this study a novel modelling of FS: we include reliability as the third objective of the problem. In order to address the modified problem, we propose the application of the non-dominated sorting genetic algorithm-III (NSGA-III). We selected six incomplete data sets from the University of California Irvine (UCI) machine learning repository. We used the mean imputation method to deal with the missing data. In the experiments, k-nearest neighbors (K-NN) is used as the classifier to evaluate the feature subsets. Experimental results show that the proposed three-objective model coupled with NSGA-III efficiently addresses the FS problem for the six data sets included in this study.

An Empirical Comparison of the Sales Forecasting Performance for Plastic Tray Manufacturing Using Missing Data

The problem of missing data is frequently met in time series analysis. If not appropriately addressed, it usually leads to failed modeling and distorted forecasting. To deal with high market uncertainty, companies need a reliable and sustainable forecasting mechanism. In this article, two propositions are presented: (1) a dedicated time series forecasting scheme, which is both accurate and sustainable, and (2) a practical observation of the data background to deal with the problem of missing data and to effectively formulate correction strategies after predictions. In the empirical study, actual tray sales data and a comparison of different models that combine missing data processing methods and forecasters are employed. The results show that a specific product needs to be represented by a dedicated model. For example, regardless of whether the last fiscal year was a growth or recession year, the results suggest that the missing data for products with a high market share should be handled by the zero-filling method, whereas the mean imputation method should be for the average market share products. Finally, the gap between forecast and actual demand is bridged by employing a validation set, and it is further used for formulating correction strategies regarding production volumes.

Bigotry in Last Observation Carried Forward (LOCF) Analysis

Last observation carried forward is a common statistical approach to the analysis of longitudinal repeated measures data where some follow-up observations may be missing. To accurately estimate the magnitude of treatment efficacy, caution should be observed regarding the LOCF analytical bias. To conciliate with this challenge, several imputation methods have been developed in the literature to handle missing values where the most commonly used are complete case method, mean imputation method, last observation carried forward (LOCF) method and multiple imputation (MI) method. Subsequently after rigorous review, this paper concludes that LOCF method has more bias than the other three methods in most situations.