Missing Data Imputation Research Articles

Missing data imputation strategies: A comparative analysis of hourly and daily pollution models

Epidemiological studies have consistently demonstrated the association between daily pollution exposure and disease outcomes. To estimate daily exposure, hourly pollution data are commonly aggregated, but missing data pose a significant challenge to this approach. To overcome this issue, some researchers have developed various models to impute missing hourly data. Alternatively, directly modelling pollution exposure on a daily basis is possible, thereby avoiding the computational burden of hourly pollution modelling. However, the performance of these two modelling strategies remains unclear. This study conducts a comparative assessment between hourly and daily modelling strategies for the purpose of estimating daily pollution exposure. Utilizing data derived from Guangzhou city, the analysis encompasses diverse scenarios of data absence. The outcomes consistently highlight the superior performance of daily pollution models in terms of mitigated bias and diminished root mean square error (RMSE) values.

Attention-based generative adversarial networks for aquaponics environment time series data imputation

Environmental parameter data collected by sensors for monitoring the environment of agricultural facility operations are usually incomplete due to external environmental disturbances and device failures. And the missing of collected data is completely at random. In practice, missing data could create biased estimations and make multivariate time series predictions of environmental parameters difficult, leading to imprecise environmental control. A multivariate time series imputation model based on generative adversarial networks and multi-head attention (ATTN-GAN) is proposed in this work to reducing the negative consequence of missing data. ATTN-GAN can capture the temporal and spatial correlation of time series, and has a good capacity to learn data distribution. In the downstream experiments, we used ATTN-GAN and baseline models for data imputation, and predicted the imputed data, respectively. For the imputation of missing data, over the 20%, 50% and 80% missing rate, ATTN-GAN had the lowest RMSE, 0.1593, 0.2012 and 0.2688 respectively. For water temperature prediction, data processed with ATTN-GAN over MLP, LSTM, DA-RNN prediction methods had the lowest MSE, 0.6816, 0.8375 and 0.3736 respectively. Those results revealed that ATTN-GAN outperformed all baseline models in terms of data imputation accuracy. The data processed by ATTN-GAN is the best for time series prediction.

Subthreshold micropulse laser versus standard laser for the treatment of central-involving diabetic macular oedema with central retinal thickness of

ObjectivesTo estimate the economic costs, health-related quality-of-life outcomes and cost-effectiveness of subthreshold micropulse laser (SML) versus standard laser (SL) for the treatment of diabetic macular oedema (DMO) with central retinal...

An effective imputation scheme for handling missing values in the heterogeneous dataset

<span>A high level of data quality has always been a concern for many applications based on machine learning, including clinical decision support systems, weather forecasting, traffic predictions, and many others. A very limited amount of work is devoted to exploiting the missing values for effective imputation and better prediction. This paper introduces a unique approach to predicting and imputing missing data fields in the multivariate dataset such as numerical, categorical, and unstructured. The proposed imputation method is a multi-model scheme based on the joint approach of natural language processing (NLP) encoders, machine learning-driven feature extractors, and a sequential regression imputation technique to predict missing values. <br /> The proposed system is robust and scalable without requiring extensive engineering. The validation of the model is done on the benchmarked clinical dataset of heart disease obtained from UCI. The results show that the proposed methods achieve better imputation accuracy and require significantly less time than other missing data imputation methods.</span>

Deep learning versus conventional methods for missing data imputation: A review and comparative study

Deep learning versus conventional methods for missing data imputation: A review and comparative study

Income Inequality for Employees with Higher Education During the Pandemic

The COVID-19 pandemic affected employability in different measures, especially workers with a basic educational level and women, contrary to this, employment increased for workers with a higher level, the increase being greater for women. In this study we will analyze how the pandemic affected workers with higher education, identifying inequality by sex, age group and professional training, in addition to measuring the impact on the income of the Economically Active Population (EAP). The data for the analysis comes from the National Occupation and Employment Survey (ENOE), for the years 2019 to 2021. The methodology used is the technique of random imputation of missing data (MAR), the Gini inequality index (IG), and the Mincer income equation. The main results show that the inequality between men and women decreased from 3% to 1.2% in the period of the beginning of the confinement, in addition it is distinguished that it favors women with higher level studies. Regarding age, the gap grew as the age of the interviewee increased; On the other hand, for workers with higher education, inequality decreased for those with a master's degree. Finally, in terms of admission, during the course of the pandemic as in the areas of Humanities, Social Sciences and Behavioral Studies, Information Sciences, Business and Administration, Mathematics and Statistics, and Veterinary Medicine, admission was better for women.

Five-year Maintenance of Clinical Response and Consistent Safety Profile for Guselkumab in Asian patients with Psoriasis from VOYAGE 1 and VOYAGE 2.

Guselkumab is a human monoclonal antibody against IL-23 used in the treatment of moderate-to-severe psoriasis. This post-hoc analysis evaluated the efficacy and safety of guselkumab in the Asian subpopulation of VOYAGE 1 and VOYAGE 2 through 5years. The proportions of guselkumab-treated Asian patients (VOYAGE 1 and 2) achieving Psoriasis Area and Severity Index (PASI) 90 and PASI 100, Investigator's Global Assessment (IGA) scores of 0/1 and 0, and Dermatology Life Quality Index (DLQI) scores of 0/1 (week 100 through week 252) were assessed. Non-responders were patients who met the treatment failure rules. Efficacy endpoints were analyzed using the as-observed methodology (no missing data imputation) for both studies and using non-responder imputation (for patients with any missing data) in VOYAGE 1. Safety outcomes were based on pooled data through week 252. Response rates through week 252 for 199 Asian patients in the guselkumab group in VOYAGE 1 and VOYAGE 2, respectively, were 76.8% and 80.6% (PASI 90), 26.8% and 38.7% (PASI 100), 64.3% and 87.1% (IGA 0/1), and 26.8% and 45.2% (IGA 0). DLQI (0/1) at week 252 was achieved by 52.7% of patients in VOYAGE 1 and 61.3% in VOYAGE 2, while DLQI (0) at week 252 was achieved by 32.7% of patients in VOYAGE 1 and 40.3% in VOYAGE 2. The safety profile was similar to the global population and remained consistent through 5years. Asian patients were followed for a total of 814 patient-years (PY). Over 85% of the guselkumab-treated patients continued treatment through week 264. The rate of serious adverse events (AEs) at week 252 was 3.07/100 PY. Rates of AEs of interest were low: serious infections, 0.74/100 PY; nonmelanoma skin cancer (NMSC), no patients; malignancies other than NMSC, 0.12/100 PY; and no major adverse cardiovascular events (MACE). These analyses confirm a continuous response over 5years, indicating that guselkumab shows therapeutic longevity in Asian patients requiring long-term treatment for moderate-to-severe psoriasis. ClinicalTrials.gov identifiers: VOYAGE 1 [NCT02207231] and VOYAGE 2 [NCT02207244].

Enhanced data imputation framework for bridge health monitoring using Wasserstein generative adversarial networks with gradient penalty

The availability of complete data is essential for accurately assessing structural stability and condition in structural health monitoring (SHM) systems. Unfortunately, data missing is a common occurrence in daily monitoring operations, which hinders real-time analysis and evaluation of structural conditions. Although considerable research has been conducted to efficiently recover missing data, the implementation of these recovery methods often encounters issues such as serious mode collapse and gradient vanishing. To address these challenges, this paper proposes a missing data imputation framework called WGAIN-GP based on Wasserstein Generative Adversarial Network with Gradient Penalty. This framework aims to enhance the stability and convergence rate of the network during the missing data recovery process. The effectiveness and robustness of the proposed method are extensively evaluated using measured acceleration data from a long-span highway-railway dual-purpose bridge. The results of the implementation demonstrate that the proposed method achieves superior recovery performance even under various missing data conditions, including high missing rates of up to 90%. Furthermore, the generality of the method is validated by successfully recovering data from different missing sensors. Additionally, the recovered data is utilized for modal analysis of the bridge's structural state, further verifying the reliability of the recovery method. The proposed recovery method offers several advantages, with its stability and robustness being particularly noteworthy. By significantly enhancing the reliability of the recovered data, this method contributes to improving the overall accuracy and effectiveness of structural health monitoring systems.

"{"Imputation of missing data in long term air pollution exposure"}"

"{"Imputation of missing data in long term air pollution exposure"}"

Multiple imputation of missing data under missing at random: including a collider as an auxiliary variable in the imputation model can induce bias.

Epidemiological studies often have missing data, which are commonly handled by multiple imputation (MI). In MI, in addition to those required for the substantive analysis, imputation models often include other variables ("auxiliary variables"). Auxiliary variables that predict the partially observed variables can reduce the standard error (SE) of the MI estimator and, if they also predict the probability that data are missing, reduce bias due to data being missing not at random. However, guidance for choosing auxiliary variables is lacking. We examine the consequences of a poorly chosen auxiliary variable: if it shares a common cause with the partially observed variable and the probability that it is missing (i.e., it is a "collider"), its inclusion can induce bias in the MI estimator and may increase the SE. We quantify, both algebraically and by simulation, the magnitude of bias and SE when either the exposure or outcome is incomplete. When the substantive analysis outcome is partially observed, the bias can be substantial, relative to the magnitude of the exposure coefficient. In settings in which a complete records analysis is valid, the bias is smaller when the exposure is partially observed. However, bias can be larger if the outcome also causes missingness in the exposure. When using MI, it is important to examine, through a combination of data exploration and considering plausible casual diagrams and missingness mechanisms, whether potential auxiliary variables are colliders.

Quality and Sensitivity of Composite Indicators for Sustainable Development

Composite indicators can be understood as a summary of well-chosen and relevant sub-indicators which are combined into a single number. Their aim is to represent a multi-dimensional construct and map the performance of entities such as countries or companies. These multidimensional constructs are for example sustainability, poverty or well-being. Composite indicators are widely applied in various disciplines such as social or economic research and benefit from their apparent ease of interpretation. In the context of the Sustainable Development Framework a composite indicator over all 17 sustainable development goals, as been proposed. As composite indicators are commonly applied in highly sensitive areas this, urges the need to discuss methodical advantages and disadvantages as well as their adequacy for performance comparisons. In this paper we discuss and illustrate quality issues with regard to aspects of the subjective choices made in the construction process of composite indicators, imputation of missing data and the survey design. As an example we construct a composite indicator on sustainable economic development using data of the Sustainable Development framework. Furthermore, we exemplify and discuss strategies and methods for the quality assessment of a composite indicator.

A generic sparse regression imputation method for time series and tabular data

Although many missing data imputation methods have been proposed in the relevant literature, they focus on either time series or tabular data, but not on both. Hence, a generic sparse regression method for missing data imputation is proposed. The imputed values of a target feature are generated by solving a sparse least squares problem using a preconditioned iterative method based on generic approximate sparse pseudoinverse. Sparsity is introduced by dummy encoding existing or constructed (through discretization) categorical features. Extensive experiments were conducted on several datasets, and the results demonstrate the effectiveness of the method for both time series and tabular data.

A missing manufacturing process data imputation framework for nonlinear dynamic soft sensor modeling and its application

Data-driven soft sensors have been widely used in manufacturing processes for product quality prediction. However, in engineering practice, traditional linear soft sensors are considered to be insufficient when the manufacturing processes present high dimensionality as well as strong nonlinearity and dynamics. Moreover, due to the economic and technical limitations, missing data problem is widespread, which affects the generalization and accuracy of soft sensors. To overcome these issues, a missing manufacturing process data imputation framework is presented for nonlinear dynamic soft sensor modeling with the purpose of quality prediction. Specifically, the generate adversarial imputation network is introduced for data imputation, and the loss functions of discriminator and generator are reasonably designed. Subsequently, a mix gated unit combining bidirectional gated recurrent unit with bidirectional minimal gated unit is proposed for nonlinear dynamic soft sensor modeling aiming at improving the performance of quality prediction. Finally, the efficiency and flexibility of the proposed framework are demonstrated through a representative manufacturing process, the hot rolling process, in which the percentages of missing data are simulated from 20% to 80%. The ideal performance indicators, root mean square error and mean absolute error, have been obtained. It can be shown that the proposed framework can provide better prediction accuracy than the competitive methods in each scenario.

Graph Machine Learning for Improved Imputation of Missing Tropospheric Ozone Data.

Gaps in the measurement series of atmospheric pollutants can impede the reliable assessment of their impacts and trends. We propose a new method for missing data imputation of the air pollutant tropospheric ozone by using the graph machine learning algorithm "correct and smooth". This algorithm uses auxiliary data that characterize the measurement location and, in addition, ozone observations at neighboring sites to improve the imputations of simple statistical and machine learning models. We apply our method to data from 278 stations of the year 2011 of the German Environment Agency (Umweltbundesamt - UBA) monitoring network. The preliminary version of these data exhibits three gap patterns: shorter gaps in the range of hours, longer gaps of up to several months in length, and gaps occurring at multiple stations at once. For short gaps of up to 5 h, linear interpolation is most accurate. Longer gaps at single stations are most effectively imputed by a random forest in connection with the correct and smooth. For longer gaps at multiple stations, the correct and smooth algorithm improved the random forest despite a lack of data in the neighborhood of the missing values. We therefore suggest a hybrid of linear interpolation and graph machine learning for the imputation of tropospheric ozone time series.

Machine-Learning-Based Imputation Method for Filling Missing Values in Ground Meteorological Observation Data

Ground meteorological observation data (GMOD) are the core of research on earth-related disciplines and an important reference for societal production and life. Unfortunately, due to operational issues or equipment failures, missing values may occur in GMOD. Hence, the imputation of missing data is a prevalent issue during the pre-processing of GMOD. Although a large number of machine-learning methods have been applied to the field of meteorological missing value imputation and have achieved good results, they are usually aimed at specific meteorological elements, and few studies discuss imputation when multiple elements are randomly missing in the dataset. This paper designed a machine-learning-based multidimensional meteorological data imputation framework (MMDIF), which can use the predictions of machine-learning methods to impute the GMOD with random missing values in multiple attributes, and tested the effectiveness of 20 machine-learning methods on imputing missing values within 124 meteorological stations across six different climatic regions based on the MMDIF. The results show that MMDIF-RF was the most effective missing value imputation method; it is better than other methods for imputing 11 types of hourly meteorological elements. Although this paper applied MMDIF to the imputation of missing values in meteorological data, the method can also provide guidance for dataset reconstruction in other industries.

One Model to Synthesize Them All: Multi-Contrast Multi-Scale Transformer for Missing Data Imputation.

Multi-contrast magnetic resonance imaging (MRI) is widely used in clinical practice as each contrast provides complementary information. However, the availability of each imaging contrast may vary amongst patients, which poses challenges to radiologists and automated image analysis algorithms. A general approach for tackling this problem is missing data imputation, which aims to synthesize the missing contrasts from existing ones. While several convolutional neural networks (CNN) based algorithms have been proposed, they suffer from the fundamental limitations of CNN models, such as the requirement for fixed numbers of input and output channels, the inability to capture long-range dependencies, and the lack of interpretability. In this work, we formulate missing data imputation as a sequence-to-sequence learning problem and propose a multi-contrast multi-scale Transformer (MMT), which can take any subset of input contrasts and synthesize those that are missing. MMT consists of a multi-scale Transformer encoder that builds hierarchical representations of inputs combined with a multi-scale Transformer decoder that generates the outputs in a coarse-to-fine fashion. The proposed multi-contrast Swin Transformer blocks can efficiently capture intra- and inter-contrast dependencies for accurate image synthesis. Moreover, MMT is inherently interpretable as it allows us to understand the importance of each input contrast in different regions by analyzing the in-built attention maps of Transformer blocks in the decoder. Extensive experiments on two large-scale multi-contrast MRI datasets demonstrate that MMT outperforms the state-of-the-art methods quantitatively and qualitatively.

A Warm Air Blanket is Superior to a Heated Mattress in Preventing Perioperative Hypothermia in Orthopedic Arthroplasties, a Time-Series Analyses

BackgroundHypothermia is a common perioperative problem that can lead to severe complications. We evaluated whether a heated mattress (HM) is superior to a warm air blanket (WA) in preventing perioperative hypothermia in patients undergoing total hip arthroplasty (THA) or total knee arthroplasty (TKA). MethodsA retrospective cohort study was performed in a teaching hospital and data were collected for all patients undergoing THA or TKA between January 1, 2015 and May 1, 2022. We used logistic and linear regressions to analyze hypothermia occurrence and important complications. Results were adjusted for confounders and time, and was present in all subgroups and after imputation of missing data. ResultsIn total, 4,683 of 5,497 patients had information on type of heating. We found more perioperative hypothermia in patients treated with an HM compared to a WA for both THA (odds ratio-adjusted 1.42 [1.0 to 1.6] P = .06) and TKA (odds ratio-adjusted 2.10 [1.5 to 3.0] P < .01). There was no difference in postoperative infections between groups (all between 0.5% and 1.3%). Patients who had an HM significantly stayed longer in the postoperative ward (a mean difference of 4 [TKA] to 6 [THA] minutes, P < .01), but there was no difference in hospital stay. ConclusionA WA is superior compared to an HM in preventing perioperative hypothermia, with no increased risk of complications. Patients who have an HM stayed longer at the postoperative ward, potentially because of higher hypothermia rates. Therefore, it is suggested to use a WA instead of an HM.

Trend analysis of malaria in urban settings in Ethiopia from 2014 to 2019

BackgroundUrbanization generally improves health outcomes of residents and is one of the potential factors that might contribute to reducing malaria transmission. However, the expansion of Anopheles stephensi, an urban malaria vector, poses a threat for malaria control and elimination efforts in Africa. In this paper, malaria trends in urban settings in Ethiopia from 2014 to 2019 are reported with a focus on towns and cities where An. stephensi surveys were conducted.MethodsA retrospective study was conducted to determine malaria trends in urban districts using passive surveillance data collected at health facilities from 2014 to 2019. Data from 25 towns surveyed for An. stephensi were used in malaria trend analysis. Robust linear models were used to identify outliers and impute missing and anomalous data. The seasonal Mann-Kendal test was used to test for monotonic increasing or decreasing trends.ResultsA total of 9,468,970 malaria cases were reported between 2014 and 2019 through the Public Health Emergency Management (PHEM) system. Of these, 1.45 million (15.3%) cases were reported from urban settings. The incidence of malaria declined by 62% between 2014 and 2018. In 2019, the incidence increased to 15 per 1000 population from 11 to 1000 in 2018. Both confirmed (microscopy or RDT) Plasmodium falciparum (67%) and Plasmodium vivax (28%) were reported with a higher proportion of P. vivax infections in urban areas. In 2019, An. stephensi was detected in 17 towns where more than 19,804 malaria cases were reported, with most of the cases (56%) being P. falciparum. Trend analysis revealed that malaria cases increased in five towns in Afar and Somali administrative regions, decreased in nine towns, and had no obvious trend in the remaining three towns.ConclusionThe contribution of malaria in urban settings is not negligible in Ethiopia. With the rapid expansion of An. stephensi in the country, the receptivity is likely to be higher for malaria. Although the evidence presented in this study does not demonstrate a direct linkage between An. stephensi detection and an increase in urban malaria throughout the country, An. stephensi might contribute to an increase in malaria unless control measures are implemented as soon as possible. Targeted surveillance and effective response are needed to assess the contribution of this vector to malaria transmission and curb potential outbreaks.

Analysis of Machine Learning Based Imputation of Missing Data

Data analysis and classification can be affected by the availability of missing data in datasets. To deal with missing data, either deletion- or imputation-based methods are used that result in the reduction of data records or imputation of incorrect predicted value. Quality of imputed data can be significantly improved if missing values are generated accurately using machine learning algorithms. In this work, an analysis of machine learning-based algorithms for missing data imputation is performed. The K-nearest neighbors (KNN) and Sequential KNN (SKNN) algorithms are used to impute missing values in datasets using machine learning. Missing values handled using a statistical deletion approach (List-wise Deletion (LD)) and ML-based imputation methods (KNN and SKNN) are then tested and compared using different ML classifiers (Support Vector Machine and Decision Tree) to evaluate the effectiveness of imputed data. The used algorithms are compared in terms of accuracy, and results yielded that the ML-based imputation method (SKNN) outperforms the LD-based approach and KNN method in terms of the effectiveness of handling missing data in almost every dataset with both classification algorithms (SVM and DT).

K Nearest Neighbour and Improved Artificial Neural Network Techiniques for Student Academic Performance Prediction

Educational Data Mining has been an emerging topic nowadays due to the growth of educational data. This field makes it possible to develop methods in order to find out hidden patterns from educational data. The methods extracted from Educational Data Mining discipline are then used to understand students including their learning behavior as well as to predict their academic performance. In recent work used artificial neural network to predict academic performance. However in recent work does not used any proper method for missing value imputation and this will affect prediction performance. To mitigate the above mentioned issue in this work introduced an improved framework for student academic performance prediction. In which pre-processing is done by using data formatting, K Nearest Neighbour (KNN) based missing data imputation, min max normalization based data normalization and data filtering methods. Once the data is enhanced using pre-processing, it will be send to the classifier for Student academic performance prediction which is done by using Improved Artificial Neural Network (IANN) method. Experimental results demonstrate the effectiveness of the proposed model interms of precision, recall and f-measure.