Missing Patterns Research Articles

Multi-Feature Map Integrated Attention Model for Early Prediction of Type 2 Diabetes Using Irregular Health Examination Records.

Type 2 diabetes (T2D) is a worldwide chronic disease that is difficult to cure and causes a heavy social burden. Early prediction of T2D can effectively identify high-risk populations and facilitate earlier implementation of appropriate preventive interventions. Various early prediction models for T2D have been proposed. However, these methods do not consider the following factors: 1) health examination records (HER) containing health information before diagnosis; 2) rating information containing clinical knowledge; and 3) local and global information of time-series features. These diagnostically relevant factors need to be considered. It is challenging due to irregular and multivariate time series. In this paper, we propose the multi-feature map integrated attention model (MFMAM) for early diabetes prediction using HER. Specifically, HER is converted into the multi-feature map to capture local and global volatility, as well as the sequence order of high-dimensional features. We concatenate rating indicators to introduce clinical knowledge. In addition, considering missing and temporal patterns, we utilize missing and time embedding to learn the complex transition of health status. We adopt attention mechanisms to capture essential features (channels) and time points (spatial). To evaluate the proposed model, we conducted experiments on real-world long-term HER. The results demonstrated that MFMAM outperformed baseline models on tasks of varying sequence lengths and prediction windows. Moreover, we applied our designs to baseline models, and their performance was considerably improved. The proposed model contributes to the short-term and long-term early prediction of T2D in individuals with varying information richness.

Networked Time-series Prediction with Incomplete Data via Generative Adversarial Network

A networked time series (NETS) is a family of time series on a given graph, one for each node. It has a wide range of applications from intelligent transportation to environment monitoring to smart grid management. An important task in such applications is to predict the future values of a NETS based on its historical values and the underlying graph. Most existing methods require complete data for training. However, in real-world scenarios, it is not uncommon to have missing data due to sensor malfunction, incomplete sensing coverage, and so on. In this article, we study the problem of NETS prediction with incomplete data . We propose networked time series Imputation Generative Adversarial Network (NETS-ImpGAN), a novel deep learning framework that can be trained on incomplete data with missing values in both history and future. Furthermore, we propose Graph Temporal Attention Networks , which incorporate the attention mechanism to capture both inter-time series and temporal correlations. We conduct extensive experiments on four real-world datasets under different missing patterns and missing rates. The experimental results show that NETS-ImpGAN outperforms existing methods, reducing the Mean Absolute Error by up to 25%.

Studying missingness in spinal cord injury data: challenges and impact of data imputation

BackgroundIn the last decades, medical research fields studying rare conditions such as spinal cord injury (SCI) have made extensive efforts to collect large-scale data. However, most analysis methods rely on complete data. This is particularly troublesome when studying clinical data as they are prone to missingness. Often, researchers mitigate this problem by removing patients with missing data from the analyses. Less commonly, imputation methods to infer likely values are applied.ObjectiveOur objective was to study how handling missing data influences the results reported, taking the example of SCI registries. We aimed to raise awareness on the effects of missing data and provide guidelines to be applied for future research projects, in SCI research and beyond.MethodsUsing the Sygen clinical trial data (n = 797), we analyzed the impact of the type of variable in which data is missing, the pattern according to which data is missing, and the imputation strategy (e.g. mean imputation, last observation carried forward, multiple imputation).ResultsOur simulations show that mean imputation may lead to results strongly deviating from the underlying expected results. For repeated measures missing at late stages (> = 6 months after injury in this simulation study), carrying the last observation forward seems the preferable option for the imputation. This simulation study could show that a one-size-fit-all imputation strategy falls short in SCI data sets.ConclusionsData-tailored imputation strategies are required (e.g., characterisation of the missingness pattern, last observation carried forward for repeated measures evolving to a plateau over time). Therefore, systematically reporting the extent, kind and decisions made regarding missing data will be essential to improve the interpretation, transparency, and reproducibility of the research presented.

Missing Data Imputation Using Socio-hawk Optimization-based Deep Neural Network

Missing data remain the common issue experienced in the real-world environment, which leads to deviation in data analysis and mining. Therefore, in order to lessen the consequences of missing data caused by human mistake, missing data imputation must be used in data processing. The traditional imputation model fails to satisfy the evaluation requirement due to its poor stability and low accuracy. Further, these models compromise the imputation accuracy of the increasing number of missing information. Hence, in this research, an optimized missing data imputation model is proposed using the Socio-hawk optimization Deep Neural Network (DNN). In this research, the DNN extracts the important features from the data, in which the missing data are estimated with an arbitrary missing pattern. It is stated that whenever the hyperparameters are tuned properly, the DNN’s performance is improved. The key here is the efficient training of DNN using the suggested Socio-hawk optimization, which improves the imputation model’s accuracy. To determine how well the suggested imputation model imputes missing data, it is compared to other methods. As a result, the paper’s primary contribution is to effectively train DNN using the suggested Socio-hawk optimization that reduces the error rate of the imputation model. The experimental evaluation shows that the proposed missing data imputation model attains a high performance at 90%, which provides 1.0595, 1.9919, and 0.9421 of MAE, MSE, and MAPE.

Smdi: an R package to perform structural missing data investigations on partially observed confounders in real-world evidence studies.

Partially observed confounder data pose a major challenge in statistical analyses aimed to inform causal inference using electronic health records (EHRs). While analytic approaches such as imputation are available, assumptions on underlying missingness patterns and mechanisms must be verified. We aimed to develop a toolkit to streamline missing data diagnostics to guide choice of analytic approaches based on meeting necessary assumptions. We developed the smdi (structural missing data investigations) R package based on results of a previous simulation study which considered structural assumptions of common missing data mechanisms in EHR. smdi enables users to run principled missing data investigations on partially observed confounders and implement functions to visualize, describe, and infer potential missingness patterns and mechanisms based on observed data. The smdi R package is freely available on CRAN and can provide valuable insights into underlying missingness patterns and mechanisms and thereby help improve the robustness of real-world evidence studies.

Understanding tourists’ intention in the face of earthquake risks: An analysis of word-of-mouth impact in the turkish tourism context

Purpose – The objective of this research is to uncover the correlation between word-of-mouth communication, electronic word-of-mouth communication, perceived risk, and the intention to visit Turkey following an earthquake that occurred in February 2023. Methodology/Design/Approach – Data was collected from adult participants aged 18 and above in the United States of America. After data collection, any missing patterns in the data records were addressed, resulting in a refined dataset consisting of 501 participants. The analysis was performed using SmartPLS-4 and SPSS. Findings – According to the results, positive word-of-mouth about earthquakes does not decrease tourists’ perceived risk of visiting Turkey. However, positive word-of-mouth and electronic word-of-mouth about earthquakes do increase tourists’ intention to visit Turkey. Positive electronic word-of-mouth about earthquakes reduces tourists’ perceived risk of visiting Turkey. Lastly, the perceived risk associated with earthquakes decreases tourists’ intention to visit Turkey. Originality of the research – In a ground-breaking research endeavour that examines the seismic events of February 2023, this study pioneers an investigation into the complex motivations of tourists who choose to visit Turkey. It unravels the intricate interplay between word-of-mouth communication, electronic word-of-mouth, perceived risk, and the inclination to visit Turkey following the earthquake

Deeply Learned Generalized Linear Models with Missing Data

Deep Learning (DL) methods have dramatically increased in popularity in recent years, with significant growth in their application to various supervised learning problems. However, the greater prevalence and complexity of missing data in such datasets present significant challenges for DL methods. Here, we provide a formal treatment of missing data in the context of deeply learned generalized linear models, a supervised DL architecture for regression and classification problems. We propose a new architecture, dlglm, that is one of the first to be able to flexibly account for both ignorable and non-ignorable patterns of missingness in input features and response at training time. We demonstrate through statistical simulation that our method outperforms existing approaches for supervised learning tasks in the presence of missing not at random (MNAR) missingness. We conclude with a case study of the Bank Marketing dataset from the UCI Machine Learning Repository, in which we predict whether clients subscribed to a product based on phone survey data. Supplementary materials for this article are available online.

Comparison of Paper-and-Pencil Versus Tablet Administration of the 2021 National Youth Risk Behavior Survey (YRBS)

PurposeAs part of efforts to modernize the Youth Risk Behavior Surveillance System, the national Youth Risk Behavior Survey (YRBS) is moving from paper-and-pencil instrument (PAPI) administration to electronic administration using tablets. This study aimed to examine differences in demographic characteristics and the reporting of health behaviors and experiences between the PAPI- and tablet-administered 2021 national YRBS questionnaire. MethodsHigh school students (grades 9–12) in classrooms from 57 schools participating in the 2021 national YRBS were assigned randomly to complete the survey using PAPI (n = 4,684 students) or using tablets (n = 3,645 students). Eighty-nine behavior and experience items were examined to compare the missingness in reporting and the prevalence estimation (i.e., proportions) by administration mode. ResultsDemographic characteristics (sex, race/ethnicity, grade, and sexual identity) did not differ by mode (PAPI vs. tablet). For the majority (93.2%, 83 out of 89) of YRBS behavior and experience items, mode was not significantly associated with the reported proportions, adjusting for sex, race/ethnicity, grade, and sexual identity. However, 30 out of 89 (33.7%) items showed significant variation in missingness by mode; 10 items had higher missingness with PAPI administration while 20 had higher missingness with tablet administration. DiscussionSurvey administration mode was not significantly associated with behavior and experience reporting among high school students. More research is needed to understand differential patterns of missingness by mode. Aligning with Centers for Disease Control and Prevention's Public Health Data Modernization Initiative, findings from this study provide evidence to support electronic survey administration for the national YRBS, particularly using tablet data collection.

UIFSS: An aid for univariate data with large missing gap in IoT applications

In the realm of Internet of Things (IoT) sensor data, missing patterns often occur due to sensor glitches and communication problems. Conventional missing data imputation methods struggle to handle multiple missing patterns, as they fail to fully leverage the available data as well as partially imputed data. To address this challenge, we propose a novel approach called Univariate data Imputation using Fast Similarity Search (UIFSS). The proposed method solved the missing data problem of IoT data using fast similarity search that can suits different patterns of missingness. Exploring similarities between data elements, a problem known as all-pairs-similarity-search, has been extensively studied in fields like text analysis. Surprisingly, applying this concept to time series subsequences hasn’t seen much progress, likely due to the complexity of the task. Even for moderately sized datasets, the traditional approach can take a long time, and common techniques to speed it up only help a bit. Notably, for very large datasets, our algorithm can be easily adapted to produce high-quality approximate results quickly. UIFSS consists of two core components:Sensor sorting with Similar Node Clustering (SSNC) and Imputation Estimator using Fast Similarity Search(IEFSS). The SSNC, encompassing missing sensor sorting depending on their entropy to guide the imputation process. Subsequently, IEFSS uses global similar sensors and captures local region volatility, prioritizing data preservation while improving accuracy through z-normalized query based similarity search. Through experiments on simulated and bench mark datasets, UIFSS outperforms existing methods across various missing patterns. This approach offers a promising solution for handling missing IoT sensor data and with improved imputation accuracy.

Dynamic assessment of scapholunate ligament status by real-time magnetic resonance imaging: an exploratory clinical study

ObjectiveClinical-standard MRI is the imaging modality of choice for the wrist, yet limited to static evaluation, thereby potentially missing dynamic instability patterns. We aimed to investigate the clinical benefit of (dynamic) real-time MRI, complemented by automatic analysis, in patients with complete or partial scapholunate ligament (SLL) tears.Material and methodsBoth wrists of ten patients with unilateral SLL tears (six partial, four complete tears) as diagnosed by clinical-standard MRI were imaged during continuous active radioulnar motion using a 1.5-T MRI scanner in combination with a custom-made motion device. Following automatic segmentation of the wrist, the scapholunate and lunotriquetral joint widths were analyzed across the entire range of motion (ROM). Mixed-effects model analysis of variance (ANOVA) followed by Tukey’s posthoc test and two-way ANOVA were used for statistical analysis.ResultsWith the increasing extent of SLL tear, the scapholunate joint widths in injured wrists were significantly larger over the entire ROM compared to those of the contralateral healthy wrists (p<0.001). Differences between partial and complete tears were most pronounced at 5°–15° ulnar abduction (p<0.001). Motion patterns and trajectories were altered. Complete SLL deficiency resulted in complex alterations of the lunotriquetral joint widths.ConclusionReal-time MRI may improve the functional diagnosis of SLL insufficiency and aid therapeutic decision-making by revealing dynamic forms of dissociative instability within the proximal carpus. Static MRI best differentiates SLL-injured wrists at 5°–15° of ulnar abduction.

The impact of imputation quality on machine learning classifiers for datasets with missing values

BackgroundClassifying samples in incomplete datasets is a common aim for machine learning practitioners, but is non-trivial. Missing data is found in most real-world datasets and these missing values are typically imputed using established methods, followed by classification of the now complete samples. The focus of the machine learning researcher is to optimise the classifier’s performance.MethodsWe utilise three simulated and three real-world clinical datasets with different feature types and missingness patterns. Initially, we evaluate how the downstream classifier performance depends on the choice of classifier and imputation methods. We employ ANOVA to quantitatively evaluate how the choice of missingness rate, imputation method, and classifier method influences the performance. Additionally, we compare commonly used methods for assessing imputation quality and introduce a class of discrepancy scores based on the sliced Wasserstein distance. We also assess the stability of the imputations and the interpretability of model built on the imputed data.ResultsThe performance of the classifier is most affected by the percentage of missingness in the test data, with a considerable performance decline observed as the test missingness rate increases. We also show that the commonly used measures for assessing imputation quality tend to lead to imputed data which poorly matches the underlying data distribution, whereas our new class of discrepancy scores performs much better on this measure. Furthermore, we show that the interpretability of classifier models trained using poorly imputed data is compromised.ConclusionsIt is imperative to consider the quality of the imputation when performing downstream classification as the effects on the classifier can be considerable.

GCN-ST-MDIR: Graph Convolutional Network-Based Spatial-Temporal Missing Air Pollution Data Pattern Identification and Recovery

Missing data pattern identification and recovery (MDIR) is vital for accurate air pollution monitoring. To recover the missing air pollution data, GCN-ST-MDIR, a Graph Convolutional Network (GCN)-based MDIR framework, is proposed to identify daily missing data patterns and automatically select the best recovery method. GCN-ST-MDIR presents four novelties: (1) A new graph construction is developed to improve GCN data representation for MDIR using S-T similarity matrix and domain-specific knowledge (e.g. weekend/weekday). (2) A TL component is used to pre-train LSCE and ILSCE models. (3) A GCN structure outputs a selection indicator to determine the dominant missing pattern for daily input. The pre-trained data recovery model's accuracy is incorporated into the GCN loss function to penalize the wrong indicator. (4) The output of the GCN structure is used as a score to combine LSCE and ILSCE. Results show that the domain-specific S-T regularity and irregularity can be used as the prior information for both GCN and ILSCE/LSCE to enhance feature extraction. Our model considerably improves the recovery performance as compared to the baselines. GCN-ST-MDIR has achieved an accuracy of 88.48% for general missing data recovery with consecutively and sporadically missing data. GCN-ST-MDIR can be extended to many other S-T MDIR challenges.

Missingness-Pattern-Adaptive Learning With Incomplete Data.

Many real-world problems deal with collections of data with missing values, e.g., RNA sequential analytics, image completion, video processing, etc. Usually, such missing data is a serious impediment to a good learning achievement. Existing methods tend to use a universal model for all incomplete data, resulting in a suboptimal model for each missingness pattern. In this paper, we present a general model for learning with incomplete data. The proposed model can be appropriately adjusted with different missingness patterns, alleviating competitions between data. Our model is based on observable features only, so it does not incur errors from data imputation. We further introduce a low-rank constraint to promote the generalization ability of our model. Analysis of the generalization error justifies our idea theoretically. In additional, a subgradient method is proposed to optimize our model with a proven convergence rate. Experiments on different types of data show that our method compares favorably with typical imputation strategies and other state-of-the-art models for incomplete data. More importantly, our method can be seamlessly incorporated into the neural networks with the best results achieved. The source code is released at https://github.com/YS-GONG/missingness-patterns.

Multiple imputation for non-monotone missing not at random data using the no self-censoring model.

Although approaches for handling missing data from longitudinal studies are well-developed when the patterns of missingness are monotone, fewer methods are available for non-monotone missingness. Moreover, the conventional missing at random assumption-a natural benchmark for monotone missingness-does not model realistic beliefs about the non-monotone missingness processes (Robins and Gill, 1997). This has provided the impetus for alternative non-monotone missing not at random mechanisms. The "no self-censoring" model is such a mechanism and assumes the probability an outcome variable is missing is independent of its value when conditioning on all other possibly missing outcome variables and their missingness indicators. As an alternative to "weighting" methods that become computationally demanding with increasing number of outcome variables, we propose a multiple imputation approach under no self-censoring. We focus on the case of binary outcomes and present results of simulation and asymptotic studies to investigate the performance of the proposed imputation approach. We describe a related approach to sensitivity analysis to departure from no self-censoring. We discuss the relationship between missing at random and no self-censoring and prove that one is not a special case of the other. Finally, we discuss extensions to non-binary data settings. The proposed methods are illustrated with application to a substance use disorder clinical trial.

An Effective Imputation Method Using Data Enrichment for Missing Data of Loop Detectors in Intelligent Traffic Control Systems

In intelligent traffic control systems, the features extracted by loop detectors are insufficient to accurately impute missing data. Most of the existing imputation methods use only these extracted features, which leads to the construction of data models that cannot fulfill the required accuracy. This deficiency is the main motivation to propose an enrichment imputation method for loop detectors namely EIM-LD, in which the imputation accuracy is increased for different missing patterns and ratios by introducing a data enrichment technique using statistical multi-class labeling. It first enriches the clean data by adding a statistical multi-class label, including C1…Cn classes. Then, the class of samples in the missed-volume data is labeled using the best data model constructed from the labeled clean data by five different classifiers. Experts of the traffic control department in Isfahan city determined classes of the statistical multi-class label for n = 5 (class labels), and we also developed subclass labels (n = 20) since the number of samples in the subclass labels was sufficient. Next, the enriched data are divided into n datasets, each of them is imputed independently using various imputation methods, and their results are finally merged. To evaluate the impact of using the proposed method, the original data, including missing volumes, are first imputed without our enrichment method. Then, the proposed method’s accuracy is evaluated by considering two class labels and subclass labels. The experimental and statistical results prove that the proposed EIM-LD method can enrich the real data collected by loop detectors, by which the comparative imputation methods construct a more accurate data model. In addition, using subclass labels further enhances the imputation method’s accuracy.

The Thoracic Research Evaluation and Treatment 2.0 Model: A Lung Cancer Prediction Model for Indeterminate Nodules Referred for Specialist Evaluation

Appropriate risk stratification of indeterminate pulmonary nodules (IPNs) is necessary to direct diagnostic evaluation. Currently available models were developed in populations with lower cancer prevalence than that seen in thoracic surgery and pulmonology clinics and usually do not allow for missing data. We updated and expanded the Thoracic Research Evaluation and Treatment (TREAT) model into a more generalized, robust approach for lung cancer prediction in patients referred for specialty evaluation. Can clinic-level differences in nodule evaluation be incorporated to improve lung cancer prediction accuracy in patients seeking immediate specialty evaluation compared with currently available models? Clinical and radiographic data on patients with IPNs from six sites (N= 1,401) were collected retrospectively and divided into groups by clinical setting: pulmonary nodule clinic (n= 374; cancer prevalence, 42%), outpatient thoracic surgery clinic (n= 553; cancer prevalence, 73%), or inpatient surgical resection (n= 474; cancer prevalence, 90%). A new prediction model was developed using a missing data-driven pattern submodel approach. Discrimination and calibration were estimated with cross-validation and were compared with the original TREAT, Mayo Clinic, Herder, and Brock models. Reclassification was assessed with bias-corrected clinical net reclassification index and reclassification plots. Two-thirds of patients had missing data; nodule growth and fluorodeoxyglucose-PET scan avidity were missing most frequently. The TREAT version 2.0 mean area under the receiver operating characteristic curve across missingness patterns was 0.85 compared with that of the original TREAT (0.80), Herder (0.73), Mayo Clinic (0.72), and Brock (0.68) models with improved calibration. The bias-corrected clinical net reclassification index was 0.23. The TREAT 2.0 model is more accurate and better calibrated for predicting lung cancer in high-risk IPNs than the Mayo, Herder, or Brock models. Nodule calculators such as TREAT 2.0 that account for varied lung cancer prevalence and that consider missing data may provide more accurate risk stratification for patients seeking evaluation at specialty nodule evaluation clinics.

IMAGE CAPTIONING: NOW EASILY DONE BY USING DEEP LEARNING MODELS

Abstract—Image captioning helps the reader to understand the context that is to be conveyed. It is highly used in the applications like finding the missing or hidden patterns, in guiding the self driving cars, in robots which are automated using Artificial Intelligence and Machine Learning. It is also used to scrutinize large amount of images. This paper includes different image captioning models by using the concept of deep learning. Here we discuss about the frameworks like CNN-CNN, CNN-RNN and some more with their implementation overview with their advantages and disadvantages

Partial replacement imputation estimation for partially linear models with complex missing pattern covariates

Partial replacement imputation estimation for partially linear models with complex missing pattern covariates

Prediction approaches for partly missing multi‐omics covariate data: A literature review and an empirical comparison study

AbstractAs the availability of omics data has increased in the last few years, more multi‐omics data have been generated, that is, high‐dimensional molecular data consisting of several types such as genomic, transcriptomic, or proteomic data, all obtained from the same patients. Such data lend themselves to being used as covariates in automatic outcome prediction because each omics type may contribute unique information, possibly improving predictions compared to using only one omics data type. Frequently, however, in the training data and the data to which automatic prediction rules should be applied, the test data, the different omics data types are not available for all patients. We refer to this type of data as block‐wise missing multi‐omics data. First, we provide a literature review on existing prediction methods applicable to such data. Subsequently, using a collection of 13 publicly available multi‐omics data sets, we compare the predictive performances of several of these approaches for different block‐wise missingness patterns. Finally, we discuss the results of this empirical comparison study and draw some tentative conclusions.This article is categorized under: Applications of Computational Statistics &gt; Genomics/Proteomics/Genetics Applications of Computational Statistics &gt; Health and Medical Data/Informatics Statistical and Graphical Methods of Data Analysis &gt; Analysis of High Dimensional Data

MSR19 Missing Patient Reported Outcome Data in Clinical Trials: An Overview and Simulation Study

Missing data occur in clinical trials and can have a negative impact on the results. In this study, we (1) outlined concepts related to missing patient reported outcomes (PRO) data, including patterns of missingness, missing data types (missing completely at random [MCAR], missing at random [MAR], missing not at random [MNAR]), and commonly used analytic strategies that address missingness (multiple imputation [MI], mixed model repeated measures [MMRM]), and (2) investigated the potential impact of missing data on PRO results in clinical trial settings.