Treatment Of Missing Data Research Articles

Risk prediction model for postoperative pneumonia in esophageal cancer patients: A systematic review.

Numerous studies have developed or validated prediction models to estimate the likelihood of postoperative pneumonia (POP) in esophageal cancer (EC) patients. The quality of these models and the evaluation of their applicability to clinical practice and future research remains unknown. This study systematically evaluated the risk of bias and applicability of risk prediction models for developing POP in patients undergoing esophageal cancer surgery. PubMed, Embase, Web of Science, Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), China National Knowledge Infrastructure (CNKI), China Science and Technology Journal Database (VIP), WanFang Database and Chinese Biomedical Literature Database were searched from inception to March 12, 2024. Two investigators independently screened the literature and extracted data. The Prediction Model Risk of Bias Assessment Tool (PROBAST) checklist was employed to evaluate both the risk of bias and applicability. A total of 14 studies involving 23 models were included. These studies were mainly published between 2014 and 2023. The applicability of all studies was good. However, all studies exhibited a high risk of bias, primarily attributed to inappropriate data sources, insufficient sample size, irrational treatment of variables and missing data, and lack of model validation. The incidence of POP in patients undergoing esophageal cancer surgery ranged from 14.60% to 39.26%. The most frequently used predictors were smoking, age, chronic obstructive pulmonary disease(COPD), diabetes mellitus, and methods of thoracotomy. Inter-model discrimination ranged from 0.627 to 0.850, sensitivity ranged between 60.7% and 84.0%, and specificity ranged from 59.1% to 83.9%. In all included studies, good discrimination was reported for risk prediction models for POP in patients undergoing esophageal cancer surgery, indicating stable model performance. However, according to the PROBAST checklist, all studies had a high risk of bias. Future studies should use the predictive model assessment tool to improve study design and develop new models with larger samples and multicenter external validation. https://www.crd.york.ac.uk/prospero, identifier CRD42024527085.

Unsupervised Imputation of Non-Ignorably Missing Data Using Importance-Weighted Autoencoders

Deep Learning (DL) methods have dramatically increased in popularity in recent years. While its initial success was demonstrated in the classification and manipulation of image data, there has been significant growth in the application of DL methods to problems in the biomedical sciences. However, the greater prevalence and complexity of missing data in biomedical datasets present significant challenges for DL methods. Here, we provide a formal treatment of missing data in the context of Variational Autoencoders (VAEs), a popular unsupervised DL architecture commonly used for dimension reduction, imputation, and learning latent representations of complex data. We propose a new VAE architecture, NIMIWAE, that is one of the first to flexibly account for both ignorable and non-ignorable patterns of missingness in input features at training time. Following training, samples can be drawn from the approximate posterior distribution of the missing data can be used for multiple imputation, facilitating downstream analyses on high dimensional incomplete datasets. We demonstrate through statistical simulation that our method outperforms existing approaches for unsupervised learning tasks and imputation accuracy. We conclude with a case study of an EHR dataset pertaining to 12,000 ICU patients containing a large number of diagnostic measurements and clinical outcomes, where many features are only partially observed.

How To Treat Missing Data In Survey Research

ABSTRACT Missing data is a pervasive problem in many research contexts, including survey research. Resulting analyses are dependent on data quality and which methods are used to deal with missing values. In this article, the author offers guidance on some of the various deletion and imputation methods available for missing data. First, the existing research on missing data treatment is reviewed. Then, common reasons for missing values are discussed. This article contributes to the literature on missing data by testing several deletion and imputation methods using two simulated datasets. The simulations illustrate how and where bias may be introduced into the data. Further, the simulations demonstrate how results and inferences may change depending on the method used. The article concludes by discussing implications for research efforts. Thus, this article offers practical guidance for dealing with missing data to both marketing academics and practitioners.

A Note on Ising Network Analysis with Missing Data.

The Ising model has become a popular psychometric model for analyzing item response data. The statistical inference of the Ising model is typically carried out via a pseudo-likelihood, as the standard likelihood approach suffers from a high computational cost when there are many variables (i.e., items). Unfortunately, the presence of missing values can hinder the use of pseudo-likelihood, and a listwise deletion approach for missing data treatment may introduce a substantial bias into the estimation and sometimes yield misleading interpretations. This paper proposes a conditional Bayesian framework for Ising network analysis with missing data, which integrates a pseudo-likelihood approach with iterative data imputation. An asymptotic theory is established for the method. Furthermore, a computationally efficient Pólya-Gamma data augmentation procedure is proposed to streamline the sampling of model parameters. The method's performance is shown through simulations and a real-world application to data on major depressive and generalized anxiety disorders from the National Epidemiological Survey on Alcohol and Related Conditions (NESARC).

Cost‐sensitive classification with time constraint on incomplete data

AbstractMissing values are common, but dealing with them by inappropriate method may lead to large classification errors. Empirical evidences show that the tree‐based classification algorithms such as classification and regression tree (CART) can benefit from imputation, especially multiple imputation. Nevertheless, less attention has been paid to incorporating multiple imputation into cost‐sensitive decision tree induction. This study focuses on the treatment of missing data based on a time‐constrained minimal‐cost tree algorithm. We introduce various approaches to handle incomplete data into the algorithm including complete‐case analysis, missing‐value branch, single imputation, feature acquisition, and multiple imputation. A simulation study under different scenarios examines the predictive performances of the proposed strategies. The simulation results show that the combination of the algorithm with multiple imputation can assure classification accuracy under the budget. A real medical data example provides insights into the problem of missing values in cost‐sensitive learning and the advantages of the proposed methods.

Machine Learning for Polymer Design to Enhance Pervaporation-Based Organic Recovery.

Pervaporation (PV) is an effective membrane separation process for organic dehydration, recovery, and upgrading. However, it is crucial to improve membrane materials beyond the current permeability-selectivity trade-off. In this research, we introduce machine learning (ML) models to identify high-potential polymers, greatly improving the efficiency and reducing cost compared to conventional trial-and-error approach. We utilized the largest PV data set to date and incorporated polymer fingerprints and features, including membrane structure, operating conditions, and solute properties. Dimensionality reduction, missing data treatment, seed randomness, and data leakage management were employed to ensure model robustness. The optimized LightGBM models achieved RMSE of 0.447 and 0.360 for separation factor and total flux, respectively (logarithmic scale). Screening approximately 1 million hypothetical polymers with ML models resulted in identifying polymers with a predicted permeation separation index >30 and synthetic accessibility score <3.7 for acetic acid extraction. This study demonstrates the promise of ML to accelerate tailored membrane designs.

Multiple Imputation When Variables Exceed Observations: An Overview of Challenges and Solutions

Missing data are a prevalent problem in psychological research that can reduce statistical power and bias parameter estimates. These problems can be mostly resolved with multiple imputation, a modern missing data treatment that is increasingly used. Imputation, however, requires the number of variables to be smaller than the number of observations (i.e., non-missing values), and this number is often exceeded due to, e.g., large assessments, high missing data rates, the inclusion of variables predictive of missing values, and the inclusion of non-linear transformations. Even when the ratio of variables to observations meets the minimum requirement, convergence failure can occur in large, complex models. Specialized techniques have been developed to overcome the challenges related to having too many variables in an imputation model, but they are still relatively unknown by researchers in psychology. Accordingly, this paper presents an overview of four imputation techniques that can be used to reduce the number of predictors in an imputation model: item aggregation with scales and parcels, passive imputation, principal component analysis (PcAux) and two-fold fully conditional specification. The purpose, advantages, limitations, and applications of each method are discussed, along with recommendations and illustrative examples, with the aims of (1) understanding different imputation methods and (2) identifying methods that could be useful for one’s imputation problem.

EvoImp: Multiple Imputation of Multi-label Classification data with a genetic algorithm.

Missing data is a prevalent problem that requires attention, as most data analysis techniques are unable to handle it. This is particularly critical in Multi-Label Classification (MLC), where only a few studies have investigated missing data in this application domain. MLC differs from Single-Label Classification (SLC) by allowing an instance to be associated with multiple classes. Movie classification is a didactic example since it can be "drama" and "bibliography" simultaneously. One of the most usual missing data treatment methods is data imputation, which seeks plausible values to fill in the missing ones. In this scenario, we propose a novel imputation method based on a multi-objective genetic algorithm for optimizing multiple data imputations called Multiple Imputation of Multi-label Classification data with a genetic algorithm, or simply EvoImp. We applied the proposed method in multi-label learning and evaluated its performance using six synthetic databases, considering various missing values distribution scenarios. The method was compared with other state-of-the-art imputation strategies, such as K-Means Imputation (KMI) and weighted K-Nearest Neighbors Imputation (WKNNI). The results proved that the proposed method outperformed the baseline in all the scenarios by achieving the best evaluation measures considering the Exact Match, Accuracy, and Hamming Loss. The superior results were constant in different dataset domains and sizes, demonstrating the EvoImp robustness. Thus, EvoImp represents a feasible solution to missing data treatment for multi-label learning.

Predicting systemic diseases in fundus images: systematic review of setting, reporting, bias, and models' clinical availability in deep learning studies.

Analyzing fundus images with deep learning techniques is promising for screening systematic diseases. However, the quality of the rapidly increasing number of studies was variable and lacked systematic evaluation. To systematically review all the articles that aimed to predict systemic parameters and conditions using fundus image and deep learning, assessing their performance, and providing suggestions that would enable translation into clinical practice. Two major electronic databases (MEDLINE and EMBASE) were searched until August 22, 2023, with keywords 'deep learning' and 'fundus'. Studies using deep learning and fundus images to predict systematic parameters were included, and assessed in four aspects: study characteristics, transparent reporting, risk of bias, and clinical availability. Transparent reporting was assessed by the TRIPOD statement, while the risk of bias was assessed by PROBAST. 4969 articles were identified through systematic research. Thirty-one articles were included in the review. A variety of vascular and non-vascular diseases can be predicted by fundus images, including diabetes and related diseases (19%), sex (22%) and age (19%). Most of the studies focused on developed countries. The models' reporting was insufficient in determining sample size and missing data treatment according to the TRIPOD. Full access to datasets and code was also under-reported. 1/31(3.2%) study was classified as having a low risk of bias overall, whereas 30/31(96.8%) were classified as having a high risk of bias according to the PROBAST. 5/31(16.1%) of studies used prospective external validation cohorts. Only two (6.4%) described the study's calibration. The number of publications by year increased significantly from 2018 to 2023. However, only two models (6.5%) were applied to the device, and no model has been applied in clinical. Deep learning fundus images have shown great potential in predicting systematic conditions in clinical situations. Further work needs to be done to improve the methodology and clinical application.

Deeply Learned Generalized Linear Models with Missing Data

Deep Learning (DL) methods have dramatically increased in popularity in recent years, with significant growth in their application to various supervised learning problems. However, the greater prevalence and complexity of missing data in such datasets present significant challenges for DL methods. Here, we provide a formal treatment of missing data in the context of deeply learned generalized linear models, a supervised DL architecture for regression and classification problems. We propose a new architecture, dlglm, that is one of the first to be able to flexibly account for both ignorable and non-ignorable patterns of missingness in input features and response at training time. We demonstrate through statistical simulation that our method outperforms existing approaches for supervised learning tasks in the presence of missing not at random (MNAR) missingness. We conclude with a case study of the Bank Marketing dataset from the UCI Machine Learning Repository, in which we predict whether clients subscribed to a product based on phone survey data. Supplementary materials for this article are available online.

Design an Optimal Decision Tree based Algorithm to Improve Model Prediction Performance

Performance of decision trees is assessed by prediction accuracy for unobserved occurrences. In order to generate optimised decision trees with high classification accuracy and smaller decision trees, this study will pre-process the data. In this study, some decision tree components are addressed and enhanced. The algorithms should produce precise and ideal decision trees in order to increase prediction performance. Additionally, it hopes to create a decision tree algorithm with a tiny global footprint and excellent forecast accuracy. The typical decision tree-based technique was created for classification purposes and is used with various kinds of uncertain information. Prior to preparing the dataset for classification, the uncertain dataset was first processed through missing data treatment and other uncertainty handling procedures to produce the balanced dataset. Three different real-time datasets, including the Titanic dataset, the PIMA Indian Diabetes dataset, and datasets relating to heart disease, have been used to test the proposed algorithm. The suggested algorithm's performance has been assessed in terms of the precision, recall, f-measure, and accuracy metrics. The outcomes of suggested decision tree and the standard decision tree have been contrasted. On all three datasets, it was found that the decision tree with Gini impurity optimization performed remarkably well.

A factored regression model for composite scores with item-level missing data.

Composite scores are an exceptionally important psychometric tool for behavioral science research applications. A prototypical example occurs with self-report data, where researchers routinely use questionnaires with multiple items that tap into different features of a target construct. Item-level missing data are endemic to composite score applications. Many studies have investigated this issue, and the near-universal theme is that item-level missing data treatment is superior because it maximizes precision and power. However, item-level missing data handling can be challenging because missing data models become very complex and suffer from the same "curse of dimensionality" problem that plagues the estimation of psychometric models. A good deal of recent missing data literature has focused on advancing factored regression specifications that use a sequence of regression models to represent the multivariate distribution of a set of incomplete variables. The purpose of this paper is to describe and evaluate a factored specification for composite scores with incomplete item responses. We used a series of computer simulations to compare the proposed approach to gold standard multiple imputation and latent variable modeling approaches. Overall, the simulation results suggest that this new approach can be very effective, even under extreme conditions where the number of items is very large (or even exceeds) the sample size. A real data analysis illustrates the application of the method using software available on the internet. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Characterization and filtering of profile and areal surface topography by combining the discrete Legendre and cosine transforms

In the standardized processing of surface topography data, the form removal and filtering operations are clearly separated. This is reflected in the current ISO standards concerning profile surface texture and areal surface texture, the ISO 21920 and the ISO 25178 series respectively. When the scale-limited surface texture is significant compared to the form to be removed, for example with additive manufactured surfaces, the dependence of the surface Fourier spectrum on the removed form and orientation may become significant. This may lead to interaction between the form removal and filtering operations. To counter this interaction, in this paper, the lower-order discrete Legendre polynomials that describe the form are combined with cosine functions that describe the surface texture. This set of base functions is orthonormalized using a Gram-Schmidt procedure. This results in a set of orthonormal functions that allow an independent parameterization of both form and texture. The concept and the related theory are given and illustrated using examples of filtering profiles and areal topographies, description of cylinders and treatment of missing data. The examples show that the concept as presented in this paper is useful for filtering surfaces with a dominant form and can be used in the parametrization of surfaces and cylindrical geometries. Also, the methods presented here can be used for filtering and parametrization in the case of missing points in the data, actual holes in the profile or non-rectangular surfaces.

Blockchain and Artificial Intelligence-based Solutions for Healthcare Management: Liver Disease Detection as a Case Study

The issue of privacy in clinical data restricts data sharing among various organizations because of legal and ethical concerns. Every medical organization (hospital, research center, testing lab, etc.) needs to protect personal and medical data privacy and confidentiality while also sharing data with efficient and accurate learning models for various diseases. This paper addresses a method that combines machine learning approaches on blockchain network for classification of patients with liver disease or not, while maintaining the privacy and enabling data sharing. The contribution of this paper is five-folds: (i) To deal with the liver disease dataset, data preprocessing phases are utilized (i.e., missing data treatment, data normalization, and data transformation). (ii) For feature reduction, two feature selection models are used: correlation feature selection (CFS) and wrapper metaheuristic (PSO+KNN). For training the proposed global model, the selected features (10, 4, 3) from the three models (all features, CFS, PSO+KNN) are distributed to the shared blockchain network. (iii) To detect liver disease in the preprocessed dataset, three machine learning methods are proposed: random forest, decision tree (J48), and Naïve Bayes. (iv) The distributed network (blockchain) will publish the results of the liver disease diagnosis on the network. (v) The smart contract shares the data and diagnosis between the medical organizations for future processing. An empirical study is executed to validate the relevance of our suggested framework for early diagnosis of liver disease using blockchain and machine learning models. The experimental results showed that feature selection using the wrapper metaheuristic (PSO+KNN) with a random forest model achieves the best performance with the minor features for detecting liver disease with an accuracy of 99.9 compared to 99.2, 89.2 for Naïve Bayes, and J48, respectively.

Data Exclusion in Policy Survey and Questionnaire Data: Aberrant Responses and Missingness

Data preprocessing is an integral step prior to analyzing data in psychological science, with implications for its potentially guiding policy. This article reports how psychological researchers address data preprocessing or quality concerns, with a focus on aberrant responses and missing data in self-report measures. 240 articles were sampled from four journals: Psychological Science, Journal of Personality and Social Psychology, Developmental Psychology, and Abnormal Psychology from 2012 to 2018. Nearly half of the studies did not report any missing data treatment (111/240; 46.25%), and if they did, the most common approach was listwise deletion (71/240; 29.6%). Studies that remove data due to missingness removed, on average, 12% of the sample. Likewise, most studies do not report any aberrant responses (194/240; 80%), but if they did, they classified 4% of the sample as suspect. Most studies are either not transparent enough about their data preprocessing steps or may be leveraging suboptimal procedures. Recommendations can improve transparency and data quality.

Handling Missing Values in Surveys With Complex Study Design: A Simulation Study

Abstract The inverse probability weighting (IPW) method is commonly used to deal with missing-at-random outcome (response) data collected by surveys with complex sampling designs. However, IPW methods generally assume that fully observed predictor variables are available for all sampled units, and it is unclear how to appropriately implement these methods when one or more independent variables are subject to missing values. Multiple imputation (MI) methods are well suited for a variety of missingness patterns but are not as easily adapted to complex sampling designs. In this case study, we consider the National Survey of Morbidity and Risk Factors (EMENO), a multistage probability sample survey. To understand the strengths and limitations of using either missing data treatment method for the EMENO, we present an extensive simulation study modeled on the EMENO health survey, with the target analysis being the estimation of population prevalence of hypertension as well as the association between hypertension and income. Both variables are subject to missingness. We test a variety of IPW and MI methods in simulation and on empirical data from the survey, assessing robustness by varying missingness mechanisms, proportions of missingness, and strengths of fitted response propensity models.

Handling Missing Data in Cross-Classified Multilevel Analyses: An Evaluation of Different Multiple Imputation Approaches

Multiple imputation (MI) is a popular method for handling missing data. In education research, it can be challenging to use MI because the data often have a clustered structure that need to be accommodated during MI. Although much research has considered applications of MI in hierarchical data, little is known about its use in cross-classified data, in which observations are clustered in multiple higher-level units simultaneously (e.g., schools and neighborhoods, transitions from primary to secondary schools). In this article, we consider several approaches to MI for cross-classified data (CC-MI), including a novel fully conditional specification approach, a joint modeling approach, and other approaches that are based on single- and two-level MI. In this context, we clarify the conditions that CC-MI methods need to fulfill to provide a suitable treatment of missing data, and we compare the approaches both from a theoretical perspective and in a simulation study. Finally, we illustrate the use of CC-MI in real data and discuss the implications of our findings for research practice.

Uncertainty in Thrifty Food Plan Cost Estimates for Community Food Security Assessments

ABSTRACT The objective of this paper is to provide insights into the impact of missing price data in food security assessments. We find that the treatment of missing data plays an important role in our understanding of the uncertainty inherent in food cost calculations. We show that imputation techniques tend to bias upwards the estimated cost of a Thrifty Food Plan basket. We find that the standard infill technique commonly employed in food security research (based on taking the average price of individual items) tends to be more biased than the experimental, multiple imputation approach we are suggesting.

Constraints to travel outside the local area: Effect on social participation and self-rated health

BackgroundTravel outside the local area allows people to meet others and access a wide selection of services and opportunities. However, travel can be constrained by the lack of good transport or by personal factors. This paper investigates whether these constraints are associated with poor self-rated health, and the extent to which the association is mediated by reduced social participation. MethodRespondents in a survey in the North of England (n = 2747) stated levels of constraints to trip frequency, number of places visited, travel distance, and travel mode for trips more than 15 miles (24 km) away from home. Path analysis tested associations between these constraints, indicators of social participation (seeing family and friends frequently and being a member of clubs or societies), and self-rated health. ResultsConstraints to the number of places visited were associated with self-rated health via reduced social participation. Constraints to trip frequency had a negative association with self-rated health via pathways other than social participation. Constraints to travel distance were not significant and constraints to car use and public transport use were associated with self-rated health via other constraints. The results varied by age group and were robust to changes in variable specification and treatment of missing data. ConclusionsThe results confirm the importance of being able to travel outside the local area for social participation and health and emphasize the need for public policies that reduce constraints to travel, by providing better conditions for the use of both private and public transport and for the realization of more frequent trips to a wider set of places.

Treatment of missing data in Bayesian network structure learning: an application to linked biomedical and social survey data

BackgroundAvailability of linked biomedical and social science data has risen dramatically in past decades, facilitating holistic and systems-based analyses. Among these, Bayesian networks have great potential to tackle complex interdisciplinary problems, because they can easily model inter-relations between variables. They work by encoding conditional independence relationships discovered via advanced inference algorithms. One challenge is dealing with missing data, ubiquitous in survey or biomedical datasets. Missing data is rarely addressed in an advanced way in Bayesian networks; the most common approach is to discard all samples containing missing measurements. This can lead to biased estimates. Here, we examine how Bayesian network structure learning can incorporate missing data.MethodsWe use a simulation approach to compare a commonly used method in frequentist statistics, multiple imputation by chained equations (MICE), with one specific for Bayesian network learning, structural expectation-maximization (SEM). We simulate multiple incomplete categorical (discrete) data sets with different missingness mechanisms, variable numbers, data amount, and missingness proportions. We evaluate performance of MICE and SEM in capturing network structure. We then apply SEM combined with community analysis to a real-world dataset of linked biomedical and social data to investigate associations between socio-demographic factors and multiple chronic conditions in the US elderly population.ResultsWe find that applying either method (MICE or SEM) provides better structure recovery than doing nothing, and SEM in general outperforms MICE. This finding is robust across missingness mechanisms, variable numbers, data amount and missingness proportions. We also find that imputed data from SEM is more accurate than from MICE. Our real-world application recovers known inter-relationships among socio-demographic factors and common multimorbidities. This network analysis also highlights potential areas of investigation, such as links between cancer and cognitive impairment and disconnect between self-assessed memory decline and standard cognitive impairment measurement.ConclusionOur simulation results suggest taking advantage of the additional information provided by network structure during SEM improves the performance of Bayesian networks; this might be especially useful for social science and other interdisciplinary analyses. Our case study show that comorbidities of different diseases interact with each other and are closely associated with socio-demographic factors.