Missing Data Imputation Research Articles

A missing data imputation method based on salp swarm algorithm for diabetes disease

Most of the medical datasets suffer from missing data, due to the expense of some tests or human faults while recording these tests. This issue affects the performance of the machine learning models because the values of some features will be missing. Therefore, there is a need for a specific type of methods for imputing these missing data. In this research, the salp swarm algorithm (SSA) is used for generating and imputing the missing values in the pain in my ass (also known Pima) Indian diabetes disease (PIDD) dataset, the proposed algorithm is called (ISSA). The obtained results showed that the classification performance of three different classifiers which are support vector machine (SVM), K-nearest neighbour (KNN), and Naïve Bayesian classifier (NBC) have been enhanced as compared to the dataset before applying the proposed method. Moreover, the results indicated that issa was performed better than the statistical imputation techniques such as deleting the samples with missing values, replacing the missing values with zeros, mean, or random values.

Evaluation of missing data imputation methods for human osteometric measurements.

It is not uncommon for biological anthropologists to analyze incomplete bioarcheological or forensic skeleton specimens. As many quantitative multivariate analyses cannot handle incomplete data, missing data imputation or estimation is a common preprocessing practice for such data. Using William W. Howells' Craniometric Data Set and the Goldman Osteometric Data Set, we evaluated the performance of multiple popular statistical methods for imputing missing metric measurements. Results indicated that multiple imputation methods outperformed single imputation methods, such as Bayesian principal component analysis (BPCA). Multiple imputation with Bayesian linear regression implemented in the R package norm2, the Expectation-Maximization (EM) with Bootstrapping algorithm implemented in Amelia, and the Predictive Mean Matching (PMM) method and several of the derivative linear regression models implemented in mice, perform well regarding accuracy, robustness, and speed. Based on the findings of this study, we suggest a practical procedure for choosing appropriate imputation methods.

English

Early heart Disease prediction can reduce heart illness, which is a leading source of worldwide death. Unhealthy eating, mental stress, genetics, a sedentary lifestyle, and other factors have all contributed to the development of heart disease. Angina pectoris, dilated cardiomyopathy, stroke, and heart disease are most commonly caused by congestive heart failure. Moreover, a precise heart disease prognosis is necessary for effective cardiac treatment. In clinical machine learning, it is dangerously challenging to predict heart disease. This paper proposes a method based on Jellyfish Search Optimization (JSO) and enabled Deep Residual Networks (DRN) to improve heart disease prediction. The input dataset is subjected to the preprocessing step, which utilizes missing data imputation and Z- score Normalization. Consequently, the pre-processed output is fed to the feature selection stage, wherein features are selected by Kumar- Hassebrook similarity. At last, in the heart disease diagnosis phase, where DRN makes the heart disease identification, DRN is trained using the proposed JSO. Moreover, the proposed DRN_JSO has effectively delivered better performance parameters with high accuracy of 84.65% F1_score of 83. 65%, Matthews’s correlation coefficient (MCC) of 85.65% and a True Positive Rate (TPR) of 82.65%.

Tile low-rank approximations of non-Gaussian space and space-time Tukey g-and-h random field likelihoods and predictions on large-scale systems

Large-scale statistical modeling has become necessary with the vast flood of geospace data coming from various sources. In space statistics, the Maximum Likelihood Estimation (MLE) is widely considered for modeling geospace data by estimating a set of statistical parameters related to a predefined covariance function. This covariance function describes the correlation between a set of geospace locations where the main goal is to model given data samples and impute missing data. Climate/weather modeling is a prevalent application for the MLE operation where data interpolation and forecasting are highly required. In the literature, the Gaussian random field is often used to describe geospace data as one of the most popular models for MLE. However, real-life datasets are often skewed and/or have extreme values, and non-Gaussian random field models are more appropriate for capturing such features. In this work, we provide an exact and approximate parallel implementation of the well-known Tukey g-and-h (TGH) non-Gaussian random field in the context of climate/weather applications. The proposed implementation alleviates the computation complexity of the log-likelihood function, which requires O(n2) storage and O(n3) operations, where N is the number of geospace locations, M is the number of time slots, and n=N×M. Based on tile low-rank (TLR) approximations, our implementation of the TGH model can tackle large-scale problems. Furthermore, we rely on task-based programming models and dynamic runtime systems to provide fast execution for the MLE operation in space and space-time cases. We assess the performance and accuracy of the proposed implementations using synthetic space and space-time datasets up to 800K. We also consider a 12-month precipitation dataset in Germany to demonstrate the advantage of using non-Gaussian over Gaussian random field models. We evaluate the prediction accuracy of the TGH model on the precipitation dataset using the Probability Integral Transformation (PIT) tool showing that the TGH model outperforms the Gaussian modeling in the real dataset. Moreover, our performance assessment indicates that TLR computations allow solving larger matrix sizes while preserving the required accuracy for prediction. The TLR-based approximation shows a speedup up to 7.29X and 2.96X over the exact solution.

Time-aware neural ordinary differential equations for incomplete time series modeling.

Internet of Things realizes the ubiquitous connection of all things, generating countless time-tagged data called time series. However, real-world time series are often plagued with missing values on account of noise or malfunctioning sensors. Existing methods for modeling such incomplete time series typically involve preprocessing steps, such as deletion or missing data imputation using statistical learning or machine learning methods. Unfortunately, these methods unavoidable destroy time information and bring error accumulation to the subsequent model. To this end, this paper introduces a novel continuous neural network architecture, named Time-aware Neural-Ordinary Differential Equations (TN-ODE), for incomplete time data modeling. The proposed method not only supports imputation missing values at arbitrary time points, but also enables multi-step prediction at desired time points. Specifically, TN-ODE employs a time-aware Long Short-Term Memory as an encoder, which effectively learns the posterior distribution from partial observed data. Additionally, the derivative of latent states is parameterized with a fully connected network, thereby enabling continuous-time latent dynamics generation. The proposed TN-ODE model is evaluated on both real-world and synthetic incomplete time-series datasets by conducting data interpolation and extrapolation tasks as well as classification task. Extensive experiments show the TN-ODE model outperforms baseline methods in terms of Mean Square Error for imputation and prediction tasks, as well as accuracy in downstream classification task.

Vaginal birth after cesarean section prediction model for Jordanian population

The rate of cesarean section has increased significantly worldwide, creating a group of women with one lower segment cesarean section concerned about the mode of delivery in their future pregnancies. This group of mothers will face a complex discussion because the likelihood for a successful vaginal birth after cesarean section provided to them is a general one. The probability of having a successful vaginal birth is the cornerstone factor of the mothers‘ decision. Therefore, providing a case-specific likelihood that respects the characteristics of each pregnancy will refine counseling, lower the decision conflict, and improve the success rate of vaginal birth trials eventually improving maternal and fetal outcomes. This paper aims to develop a clinical decision support system to evaluate the individualized likelihood mode of delivery for pregnant women with a previous lower segment cesarean section based on their unique characteristics. The study included six hundred fifty-nine pregnant women, where three hundred twenty-seven records had missing values. Various pre-processing steps, including missing data imputation and feature selection, were applied to the original dataset before model development to improve the data quality. Missing values were handled first, then a feature selection process using a genetic algorithm was applied to select the relevant features and to exclude features that may have been affected negatively by missing data imputation. After that, four machine learning classifiers, namely Decision Tree, Random Forest, K-Nearest Neighbors (KNN), and Logistic Regression, were used to build the prediction model. The results showed that imputing missing values followed by feature selection was more efficient than deleting them since the Area Under the Curve (AUC) has increased from 0.655 to 0.812 using the KNN classifier.

Comparative Study of Imputation Methods for Time Series Data

Missing and incomplete values pose a significant challenge in analyzing tabular and time-series data. Dealing with missing values is time-consuming and tedious, especially when working with data from real-world applications. While some imputation approaches estimate missing values based on existing observations, these methods often rely on strong assumptions about the data distribution, which only sometimes improves downstream accuracy. Although tabular imputation methods can be applied to time-series data, incorporating the time component can enhance accuracy. This study evaluates various techniques for missing data imputation in time-series data. We run experiments on four multi-variate time series datasets using five imputation methods. We report training time and testing accuracy.

Strategies for data normalization and missing data imputation and consequences for potential diagnostic microRNA biomarkers in epithelial ovarian cancer.

MicroRNAs (miRNAs) are small non-coding RNA molecules regulating gene expression with diagnostic potential in different diseases, including epithelial ovarian carcinomas (EOC). As only a few studies have been published on the identification of stable endogenous miRNA in EOC, there is no consensus which miRNAs should be used aiming standardization. Currently, U6-snRNA is widely adopted as a normalization control in RT-qPCR when investigating miRNAs in EOC; despite its variable expression across cancers being reported. Therefore, our goal was to compare different missing data and normalization approaches to investigate their impact on the choice of stable endogenous controls and subsequent survival analysis while performing expression analysis of miRNAs by RT-qPCR in most frequent subtype of EOC: high-grade serous carcinoma (HGSC). 40 miRNAs were included based on their potential as stable endogenous controls or as biomarkers in EOC. Following RNA extraction from formalin-fixed paraffin embedded tissues from 63 HGSC patients, RT-qPCR was performed with a custom panel covering 40 target miRNAs and 8 controls. The raw data was analyzed by applying various strategies regarding choosing stable endogenous controls (geNorm, BestKeeper, NormFinder, the comparative ΔCt method and RefFinder), missing data (single/multiple imputation), and normalization (endogenous miRNA controls, U6-snRNA or global mean). Based on our study, we propose hsa-miR-23a-3p and hsa-miR-193a-5p, but not U6-snRNA as endogenous controls in HGSC patients. Our findings are validated in two external cohorts retrieved from the NCBI Gene Expression Omnibus database. We present that the outcome of stability analysis depends on the histological composition of the cohort, and it might suggest unique pattern of miRNA stability profiles for each subtype of EOC. Moreover, our data demonstrates the challenge of miRNA data analysis by presenting various outcomes from normalization and missing data imputation strategies on survival analysis.

Evaluating serological tests for foot-and-mouth disease while accounting for different serotypes and uncertain vaccination status

Controlling foot-and-mouth disease (FMD) by vaccination requires adequate population coverage and high vaccine efficacy under field conditions. To assure veterinary services that animals have acquired sufficient immunity, strategic post-vaccination surveys can be conducted to monitor the coverage and performance of the vaccine. Correct interpretation of these serological data and an ability to derive exact prevalence estimates of antibody responses requires an awareness of the performance of serological tests. Here, we used Bayesian latent class analysis to evaluate the diagnostic sensitivity and specificity of four tests. A non-structural protein (NSP) ELISA determines vaccine independent antibodies from environmental exposure to FMD virus (FMDV), and three assays measuring total antibodies derived from vaccine antigen or environmental exposure to two serotypes (A, O): the virus neutralisation test (VNT), a solid phase competitive ELISA (SPCE), and a liquid phase blocking ELISA (LPBE). Sera (n=461) were collected by a strategic post-vaccination monitoring survey in two provinces of Southern Lao People's Democratic Republic (PDR) after a vaccination campaign in early 2017. Not all samples were tested by every assay and each serotype: VNT tested for serotype A and O, whereas SPCE and LPBE tested for serotype O, and only NSP-negative samples were tested by VNT, with 90 of them not tested (missing by study design). These data challenges required informed priors (based on expert opinion) for mitigating possible lack of model identifiability. The vaccination status of each animal, its environmental exposure to FMDV, and the indicator of successful vaccination were treated as latent (unobserved) variables. Posterior median for sensitivity and specificity of all tests were in the range of 92-99 %, except for the sensitivity of NSP (∼66%) and the specificity of LPBE (∼71 %). There was strong evidence that SPCE outperformed LPBE. In addition, the proportion of animals recorded as having been vaccinated that showed a serological immune response was estimated to be in the range of 67-86 %. The Bayesian latent class modelling framework can easily and appropriately impute missing data. It is important to use field study data as diagnostic tests are likely to perform differently on field survey samples compared to samples obtained under controlled conditions.

A Novel Two Tier Missing at Random Type Missing Data Imputation using Enhanced Linear Interpolation Technique on Internet of Medical Things

A Novel Two Tier Missing at Random Type Missing Data Imputation using Enhanced Linear Interpolation Technique on Internet of Medical Things

A Robust Deep Learning-Based Damage Identification Approach for SHM Considering Missing Data

Data-driven methods have shown promising results in structural health monitoring (SHM) applications. However, most of these approaches rely on the ideal dataset assumption and do not account for missing data, which can significantly impact their real-world performance. Missing data is a frequently encountered issue in time series data, which hinders standardized data mining and downstream tasks such as damage identification and condition assessment. While imputation approaches based on spatiotemporal relations among monitoring data have been proposed to handle this issue, they do not provide additional helpful information for downstream tasks. This paper proposes a robust deep learning-based method that unifies missing data imputation and damage identification tasks into a single framework. The proposed approach is based on a long short-term memory (LSTM) structured autoencoder (AE) framework, and missing data is simulated using the dropout mechanism by randomly dropping the input channels. Reconstruction errors serve as the loss function and damage indicator. The proposed method is validated using the quasi-static response (cable tension) of a cable-stayed bridge released in the 1st IPC-SHM, and results show that missing data imputation and damage identification can be effectively integrated into the proposed unified framework.

P178 Current prescribing practices in psoriatic arthritis - comparison between the UK and Europe

Abstract Background/Aims Psoriatic arthritis (PsA) is a multi-system disease with a range of management options. Treatment may vary by geographic location. We compared disease characteristics and prescribing practices in the UK and Europe in the post-Covid era. Methods The ASSIST study was a cross-sectional observational study of PsA patients aged 18 years and older selected from 24 centres across 5 countries (UK, France, Germany, Italy and Spain) between July 2021 and March 2022 (IRAS: 287039). Patients attending a face-to-face appointment with a diagnosis of PsA made by a rheumatologist were selected by systematic sampling at each centre and treated in routine clinical practice. Patient and disease characteristics, current treatment and treatment decisions (medications unchanged, switched, added or reduced) were recorded. The analysis was descriptive, with no imputation of missing data. Results 503 patients were included, with arthritis subtype, patient age, disease activity and duration shown (Table 1). Physician- and patient-reported disease severity was highest in the UK, where median patient age was lowest. Conventional synthetic (cs) DMARDS constituted a higher percentage of current PsA treatment in UK than continental Europe (66.4% vs 44.9%), whereas biologic use was more frequent in Europe (68.1% vs 36.4%). Adalimumab was the most commonly used biologic in the UK and Spain. Adalimumab and secukinumab were equally used in Germany, and ixekizumab and adalimumab were joint-first in Italy. Implementing change to the current PsA treatment was most common in the UK, predominantly being a treatment increase. This may reflect the higher level of disease activity or younger patient age in the UK than other countries, as treatment escalation is more likely earlier in the disease course. In the UK, treatment escalation was more commonly achieved by medication addition (26.2%) than medication switch (14%) or dose increase (7.5%). In Europe, medication addition and switch were of more similar frequency (10.9% vs 9.85%). Conclusion Disease characteristics and treatment strategies varied between countries, but particularly between UK and the rest of Europe. In contrast to mainland Europe, csDMARDs predominated in the UK, perhaps reflecting current NICE guidelines. Treatment escalation was most common in the UK, in keeping with higher disease activity. Disclosure L. Watson: None. C. Coyle: None. M. Brooke: None. U. Kiltz: None. E. Lubrano: None. R. Queiro: None. D. Trigos: None. J. Brandt-Juergens: None. S. D'Angelo: None. A. Delle Sedie : None. E. Dernis: None. P. Helliwell: None. P. Ho: None. A. Hueber: None. B. Joven: None. M. Koehm: None. C. Montilla: None. J. Packham: None. J.P. Tasende: None. F.J.R. Garcia: None. A. Ruyssen-Witrand: None. R. Scrivo: None. S. Twigg: None. M. Welcker: None. M. Soubrier: None. T. Wirth: None. L. Gossec: Consultancies; AbbVie, Amgen, BMS, Celltrion, Galapagos, Gilead, GSK, Janssen, Lilly, MSD, Novartis, Pfizer, Sandoz, UCB. Grants/research support; Amgen, Galapagos, Lilly, Pfizer, Sandoz, UCB. L.C. Coates: None.

P179 Shared decision-making in psoriatic arthritis consultations

Abstract Background/Aims A personalised approach is required to optimise management of psoriatic arthritis (PsA). Shared decision-making between physician and patient is key, resulting in greater patient satisfaction and outcomes. We assessed the degree of perceived collaboration following clinic visits in the UK and Europe and whether this was associated with treatment escalation. Methods The ASSIST study was a cross-sectional observational study of PsA patients aged 18 years and older selected from 24 centres across 5 countries (UK, France, Germany, Italy and Spain) between July 2021 and March 2022 (IRAS: 287039). Patients attending a face-to-face appointment with a diagnosis of PsA made by a rheumatologist were selected by systematic sampling at each centre and treated in routine clinical practice. Patients completed the collaboRATE questionnaire (scored 0-9), where high scores indicate greater perceived collaboration. The perceived efficacy in patient-physician interactions (PEPPI) tool (scored 5-25) assessed the patients’ view on their confidence in the consultation. Patient, physician, and disease characteristics were recorded, alongside treatment decisions (medications unchanged, switched, added or reduced). The analysis was descriptive, with no imputation of missing data. Results 503 patients were included, with key characteristics shown (Table 1). Generally, the level of disease severity was low (mean total PsAID score 3.6/10) and PEPPI scores were high, indicating patient confidence in the consultation. A subgroup (n = 10) perceived difficulty in sharing their concerns (PEPPI <12/25). However, collaboRATE scores remained high (mean score 7.96), suggesting satisfaction in the physician’s effort to understand patient concerns. Consultation satisfaction was not associated with treatment decisions: mean collaboRATE and PEPPI scores in those with and without treatment escalation were similar. Mean collaboRATE and PsAID scores were not associated. In patients with low collaboRATE scores (<5), only patients with higher PsAID scores (>5) had treatment escalation. However, in patients with high collaboRATE scores, even patients with low PsAID scores had treatment escalation. Conclusion Patients report high levels of shared decision-making in face-to-face PsA consultations, unrelated to treatment escalation. In patients with low PsAID scores, those with higher perceived collaboration are more likely to have treatment escalation than those without, perhaps reflecting the identification of otherwise undetected symptoms/concerns. Disclosure L.L. Watson: None. C. Coyle: None. M. Brooke: None. U. Kiltz: None. E. Lubrano: None. R. Queiro: None. D. Trigos: None. J. Brandt-Juergens: None. S. D'Angelo: None. A. Delle Sedie : None. E. Dernis: None. P. Helliwell: None. P. Ho: None. A. Hueber: None. B. Joven: None. M. Koehm: None. C. Montilla: None. J. Packham: None. J.P. Tasende: None. F.J.R. Garcia: None. A. Ruyssen-Witrand: None. R. Scrivo: None. S. Twigg: None. M. Welcker: None. M. Soubrier: None. T. Wirth: None. L. Gossec: Consultancies; AbbVie, Amgen, BMS, Celltrion, Galapagos, Gilead, GSK, Janssen, Lilly, MSD, Novartis, Pfizer, Sandoz, UCB. Grants/research support; Amgen, Galapagos, Lilly, Pfizer, Sandoz, UCB. L.C. Coates: None.

Predictors of recurrence of cancer-associated venous thromboembolism after discontinuation of anticoagulant therapy: a multicenter cohort study

BackgroundData on recurrence after the end of anticoagulation for a first event of cancer-associated venous thromboembolism (VTE) are scarce. ObjectivesOur aim was to assess predictors of VTE recurrence during a 1-year follow-up period. MethodsThis study is an analysis of RIETE, an international, multicenter, prospective cohort study of patients diagnosed with VTE. Patients had to have active cancer at the time of VTE and to have withdrawn from anticoagulation after 3 months of full treatment. Analyses were performed using Fine and Gray models, with death as a competing risk, and multiple imputation of missing data was performed by chained equations. ResultsAmong 14 318 patients with cancer-associated VTE, 3414 had undergone time-limited anticoagulation for at least 3 months. The cumulative incidence function for recurrent VTE was 10.2% (95% CI, 9.1-11.5) at 1 year. Chronic kidney disease (a subhazard ratio [sHR] of 1.08 for 10-mL/min decrease in glomerular filtration rate; 95% CI, 1.02-1.14); cancer of the lung, brain, stomach, esophagus, liver, or ovary (sHR, 3.56; 95% CI, 1.07-11.80; compared with cancer of the oropharynx, larynx, or melanoma); cancer of the pancreas, the biliary tract, or of unknown origin (sHR, 6.86; 95% CI, 1.89-24.85); inferior vena cava filter (sHR, 3.16; 95% CI, 1.75-5.71); postthrombotic syndrome (sHR, 2.09; 95% CI, 1.06-4.15); and residual pulmonary thrombotic obstruction (sHR, 2.58; 95% CI, 1.38-4.82) were predictive of recurrence. Surgery during the 2 months before VTE was predictive of absence of recurrence (sHR, 0.60; 95% CI, 0.40-0.92). ConclusionOne year after anticoagulant cessation for cancer-associated VTE, approximately 10% of patients experienced recurrence. Discontinuing anticoagulant therapy seems safe, mainly in surgery-associated VTE.

A Novel Method for Imputing Missing Values in Ship Static Data Based on Generative Adversarial Networks

Ship data obtained through the maritime sector will inevitably have missing values and outliers, which will adversely affect the subsequent study. Many existing methods for missing data imputation cannot meet the requirements of ship data quality, especially in cases of high missing rates. In this paper, a missing data imputation method based on generative adversarial networks (GANs) is proposed. The generative adversarial imputation network (GAIN) is improved using the Wasserstein distance and gradient penalty to handle missing values. Meanwhile, the data preprocessing process is optimized by combining knowledge from the ship domain, such as using isolation forests for anomaly detection. Statistical analysis of ship data is also conducted, including correlation analysis of ship design parameters, analysis of outliers, and analysis of missing data types. These analyses provide the basis for the proposed model. In a case study of 8167 bulk carriers, the proposed model outperformed the missing forest (MF) and polynomial fitting (PF) models, with an average error reduction of 2.4% and 6.3%, respectively. The proposed model also showed stable performance in cases of high missing rates. This study provides a new approach for estimating or imputing critical parameters of ships.

Transformer-enabled generative adversarial imputation network with selective generation (SGT-GAIN) for missing region imputation

Although data have been extensively leveraged for process monitoring and control in advanced manufacturing, it still suffers from the connection issues among sensors, machines, and computers, which may lead to significant data loss, i.e., missing region in the collected data, in the application of data-driven monitoring. To address the missing region issues, one popular way is to perform missing data imputation. With the advances of machine learning, many approaches have been developed for the missing data imputation, such as the popular Generative Adversarial Imputation Network (GAIN), which is based on the Generative Adversarial Network (GAN). However, the inherent shortcomings of generative adversarial architecture may still lead to unstable training. More importantly, the collected online sensor data in manufacturing are in sequential order whereas GAIN considered the input data independently. Hence, to address these two limitations, this work proposes a novel approach termed transformer-enabled GAIN with selective generation (SGT-GAIN). The contributions of the proposed SGT-GAIN consist of three aspects: (i) the architecture for transformer-enabled generation is developed to capture the sequential information among the data; (ii) a selective multi-generation framework is proposed to further reduce the imputation bias; and (iii) an ensemble learning framework is applied to enhance the imputation robustness. Both the numerical simulation study and a real-world case study in additive manufacturing demonstrated the effectiveness of the proposed SGT-GAIN.

Maize Yield Prediction using Artificial Neural Networks based on a Trial Network Dataset

The prediction of grain yield is important for sowing, cultivar positioning, crop management, and public policy. This study aims to predict maize productivity by applying an artificial neural network and by building models of multilayer perceptrons (MLPs) using public data and maize experimental networks. The dataset included parameters of climate, soil water balance, and agronomic characteristics from maize hybrids of an experimental network of two agricultural years. The climatic and soil balance water parameters were divided according to the maize plant development stages. Six databases were obtained by combining the imputation of missing data with the agronomic characteristics of the maize hybrids, the climatic parameters/soil water balance, and the complete database with both. Hyper parameterization of the models was obtained using GridSearch and k-fold cross-validation. The models with imputation were more accurate than those without it. The model with climate data/soil water balance and the complete model with imputation presented the smallest errors of 71 kg ha−1. In all the models, cultivars, locations, and their interactions were important, and different climatic conditions had the greatest weight in predicting productivity. It was concluded that the MLP models performed adequately and captured the non-linear effects of the interaction between the environment and maize hybrids. Climatic and soil balance water parameters at different stages of maize plant development explain the productivity of maize hybrids more than the agronomic characteristics of the cultivars.

Time Series Data Preparation for Failure Prediction in Smart Water Taps (SWT)

Smart water tap (SWT) time series model development for failure prediction requires acquiring data on the variables of interest to researchers, planners, engineers and decision makers. Thus, the data are expected to be ‘noiseless’ (i.e., without discrepancies such as missing data, data redundancy and data duplication) raw inputs for modelling and forecasting tasks. However, historical datasets acquired from the SWTs contain data discrepancies that require preparation before applying the dataset to develop a failure prediction model. This paper presents a combination of the generative adversarial network (GAN) and the bidirectional gated recurrent unit (BiGRU) techniques for missing data imputation. The GAN aids in training the SWT data trend and distribution, enabling the imputed data to be closely similar to the historical dataset. On the other hand, the BiGRU was adopted to save computational time by combining the model’s cell state and hidden state during data imputation. After data imputation there were outliers, and the exponential smoothing method was used to balance the data. The result shows that this method can be applied in time series systems to correct missing values in a dataset, thereby mitigating data noise that can lead to a biased failure prediction model. Furthermore, when evaluated using different sets of historical SWT data, the method proved reliable for missing data imputation and achieved better training time than the traditional data imputation method.

A robust predictive diagnosis model for diabetes mellitus using Shapley-incorporated machine learning algorithms

With the rapid advancement and integration of Artificial Intelligence (AI) in medicine, the need for new developments and accuracy in Machine Learning (ML) models and algorithms has increased substantially. Meanwhile, the limited availability of medical data has hampered the rapid advancement of AI, and robust models that can leverage the available data are needed. In this paper, we propose robust frameworks for the predictive diagnosis of diabetes mellitus using the limited data generated among women aged 21 to 81. The proposed frameworks have data augmentation, analysis of the attributes, and missing data imputations in common as the preliminary steps. We used Shapley Additive Explanation (SHAP) to extract feature importance and ascertain the most important features for fitting Extra Tree (ET), Random Forest (RF), Adaboost, and Xgboost models. The SHAP shows that glucose is the particular feature that contributes most to the prediction of diabetes, while in combination with age and Body Mass Index (BMI), they have a much more impact. Additionally, BMI and diabetes pedigree function also rate high for the prediction of diabetes; so, if it is hard to manage blood glucose, the focus can be switched to the management of BMI and the diabetes pedigree function. Informed by SHAP, we use a new dataset coined from the original one to fit the ML algorithms used for the prediction of diabetes, for which Xgboost and Adaboost performed better than other models with an accuracy of 94.67% each and an F1 score of 95.27 and 95.95, respectively.

Data processing pipeline for cardiogenic shock prediction using machine learning.

Recent advances in machine learning provide new possibilities to process and analyse observational patient data to predict patient outcomes. In this paper, we introduce a data processing pipeline for cardiogenic shock (CS) prediction from the MIMIC III database of intensive cardiac care unit patients with acute coronary syndrome. The ability to identify high-risk patients could possibly allow taking pre-emptive measures and thus prevent the development of CS. We mainly focus on techniques for the imputation of missing data by generating a pipeline for imputation and comparing the performance of various multivariate imputation algorithms, including k-nearest neighbours, two singular value decomposition (SVD)-based methods, and Multiple Imputation by Chained Equations. After imputation, we select the final subjects and variables from the imputed dataset and showcase the performance of the gradient-boosted framework that uses a tree-based classifier for cardiogenic shock prediction. We achieved good classification performance thanks to data cleaning and imputation (cross-validated mean area under the curve 0.805) without hyperparameter optimization. We believe our pre-processing pipeline would prove helpful also for other classification and regression experiments.