Different Types Of Missing Data Research Articles

A new multi-picture architecture for learned video deinterlacing and demosaicing with parallel deformable convolution and self-attention blocks

Despite the fact real-world video deinterlacing and demosaicing are well-suited to supervised learning from synthetically degraded data because the degradation models are known and fixed, learned video deinterlacing and demosaicing have received much less attention compared to denoising and super-resolution tasks. We propose a new multi-picture architecture for video deinterlacing or demosaicing by aligning multiple supporting pictures with missing data to a reference picture to be reconstructed, benefiting from both local and global spatio-temporal correlations in the feature space using modified deformable convolution blocks and a novel residual efficient top-k self-attention (kSA) block, respectively. Separate reconstruction blocks are used to estimate different types of missing data. Our extensive experimental results, on synthetic or real-world datasets, demonstrate that the proposed novel architecture provides superior results that significantly exceed the state-of-the-art for both tasks in terms of PSNR, SSIM, and perceptual quality. Ablation studies are provided to justify and show the benefit of each novel modification made to the deformable convolution and residual efficient kSA blocks. Code is available: https://github.com/KUIS-AI-Tekalp-Research-Group/Video-Deinterlacing.

A Hybrid Missing Data Imputation Method for Batch Process Monitoring Dataset.

Batch process monitoring datasets usually contain missing data, which decreases the performance of data-driven modeling for fault identification and optimal control. Many methods have been proposed to impute missing data; however, they do not fulfill the need for data quality, especially in sensor datasets with different types of missing data. We propose a hybrid missing data imputation method for batch process monitoring datasets with multi-type missing data. In this method, the missing data is first classified into five categories based on the continuous missing duration and the number of variables missing simultaneously. Then, different categories of missing data are step-by-step imputed considering their unique characteristics. A combination of three single-dimensional interpolation models is employed to impute transient isolated missing values. An iterative imputation based on a multivariate regression model is designed for imputing long-term missing variables, and a combination model based on single-dimensional interpolation and multivariate regression is proposed for imputing short-term missing variables. The Long Short-Term Memory (LSTM) model is utilized to impute both short-term and long-term missing samples. Finally, a series of experiments for different categories of missing data were conducted based on a real-world batch process monitoring dataset. The results demonstrate that the proposed method achieves higher imputation accuracy than other comparative methods.

How to deal with missing data? Multiple imputation by chained equations: recommendations and explanations for clinical practice

The presence of missing data, a constant problem in medical research, has several consequences: systematic loss of power, associated or not with a reduction in the representativeness of the sample analyzed. There are three types of missing data: 1) missing completely at random (MCAR); 2) missing at random (MAR); 3) missing not at random (MNAR). Multiple imputation by chained equations allows for the correct handling of missing data under the MCAR and MAR assumptions. It allows to simulate for each missing data j, a number m of simulated values which seem plausible with regard to the other variables. A random effect is included in this simulation to express the uncertainty. Several data sets are thus created and analyzed individually, in an identical way. Then the estimators of each data set are combined to obtain a global estimator. Multiple imputation increases power, corrects for some biases and has the advantage of being applicable to many types of variables. Complete case analysis should no longer be the norm. The objective of this guide is to help the reader in conducting an analysis with multiple imputed data. We cover the following points: the different types of missing data, the different historical approaches to handling them, and then we detail the multiple imputation method using chained equations. We provide a code example for the mice package of R®.

A Novel Imputation Model for Missing Concrete Dam Monitoring Data

To ensure the safety of concrete dams, a large number of monitoring instruments are embedded in the bodies and foundations of the dams. However, monitoring data are often missing due to failure of monitoring equipment, human error and other factors that cause difficulties in diagnosis of dam safety and failure to precisely predict their deformation. In this paper, a new method for imputing missing deformation data is proposed. First, since the traditional deformation increment speed distance index of the deformation similarity index does not take into account the fact that there is little change in deformations occurring in two consecutive days, the denominator of the index tends to be equal to zero. In this paper, an improved index for solving this problem is proposed. A combined weighting method for calculating the deformation similarity comprehensive index and the k-means clustering method is then proposed and used to classify deformation monitoring points. Subsequently, a panel data model that imputes different types of missing data is established. The method proposed in this paper can impute missing concrete dam deformation data more accurately; therefore, it can effectively solve the missing deformation monitoring data problem.

A competing risk joint model for dealing with different types of missing data in an intervention trial in prodromal Alzheimer’s disease

BackgroundMissing data can complicate the interpretability of a clinical trial, especially if the proportion is substantial and if there are different, potentially outcome-dependent causes.MethodsWe aimed to obtain unbiased estimates, in the presence of a high level of missing data, for the intervention effects in a prodromal Alzheimer’s disease trial: the LipiDiDiet study. We used a competing risk joint model that can simultaneously model each patient’s longitudinal outcome trajectory in combination with the timing and type of missingness.ResultsUsing the competing risk joint model, we were able to provide unbiased estimates of the intervention effects in the presence of the different types of missingness. For the LipiDiDiet study, the intervention effects remained statistically significant after this correction for the timing and type of missingness.ConclusionMissing data is a common problem in (Alzheimer) clinical trials. It is important to realize that statistical techniques make specific assumptions about the missing data mechanisms. When there are different missing data sources, a competing risk joint model is a powerful method because it can explicitly model the association between the longitudinal data and each type of missingness.Trial registrationDutch Trial Register, NTR1705. Registered on 9 March 2009

RESI: A Region-Splitting Imputation method for different types of missing data

A certain degree of data loss seriously affects the accuracy and availability of data, especially on the effects of the subsequent in-depth data analysis and mining. It is of great value in practical applications to construct a data imputation model, which is suitable for completing different types of missing data, including numerical only, categorical only and mixed-type data, and has strong capability of generalization. To address this issue, this paper defines a new metric, mean integrity rate, to measure the missing degree of a dataset, and proposes RESI, a novel tuple-based REgion-Splitting Imputation model, to impute different type missing data. We first select features and assign weights to each attribute by using the entropy weight method, and then partition the tuples into a subset of complete tuples and several subsets of incomplete tuples based on their integrity rate, which is formulated with the weights of attributes and the missing degree of tuples. The model performs training iterations on the complete tuple subset. In each iteration, the trained model is used to impute the next missing subset, and the computed subset is merged into the complete subset for training the next model. To improve the imputation accuracy, we leverage k-fold cross validation to correct errors. Besides imputing diverse types of missing data, extensive experimental results have shown that our model, RESI, significantly outperforms the state-of-the-art methods in the sensitivity to missing rate and accuracy of imputed data.

Missing Air Pollution Data Recovery Based on Long-Short Term Context Encoder

Air pollution has become a global challenge, and obtaining real-time air quality information is urgently needed. Although the governments have been trying their best in delivering accurate air quality reports, missing air pollution data remains a key challenge. Based on the temporal-spatial correlation of the data, we propose a novel long-short term context encoder (LSCE) structure for recovering missing air pollution data. The original context encoder approach based on image completion focuses on reconstructing rectangular missing regions. Differing from traditional methods, our fully convolutional neural network architecture enjoys the following novelties. First, LSCE can recover irregular missing data patterns. Second, we devise two data pre-processing strategies to produce two types of context encoders, namely, the long-short term cutting context encoder (LSCCE) and the long-short term sliding context encoder (LSSCE). Compared with LSCCE, LSSCE increases the number of training data matrixes. Finally, we investigate the significance of adaptive training in addressing different types of missing data. Our simulation results have demonstrated that our approach, especially, LSSCE, can outperform existing missing data recovery methods. Besides, our techniques can be widely applicable for recovering other temporally and spatially correlated missing data, such as vehicular traffic or meteorology data.

Comparison of maximum likelihood approach, Diggle–Kenward selection model, pattern mixture model with MAR and MNAR dropout data

In longitudinal studies, missing data are ubiquitous. This article made a comparison of three model-based techniques for handling different types of missing data (i.e., missing at random (MAR)-based maximum likelihood (ML) approach, missing not at random (MNAR)-based Diggle–Kenward (DK) selection model and MNAR-based pattern mixture (PM) model) in longitudinal studies through a Monte Carlo simulation study. Two influential factors were considered: the dropout rates (5%, 10%, 20%, and 40%) and the sample sizes (100, 300, 500, and 1000) under MAR and MNAR missingness mechanisms respectively. The results indicated that the model selection was a crucial issue when researchers were dealing with missing data in longitudinal studies because under MNAR mechanism, DK method outperformed MAR-based ML approach, but PM method performed worse than MAR-based method did. The differences of the parameter estimation among three methods became more significant as the sample size and the dropout rate increased.

An extensive analysis of the interaction between missing data types, imputation methods, and supervised classifiers

Abstract When applying data-mining techniques to real-world data, we often find ourselves facing observations that have no value recorded for some attributes. This can be caused by several phenomena, such as a machine’s incapability to record certain characteristics or a person refusing to answer a question in a poll. Depending on that motivation, values gone missing may follow one kind of pattern or another, or describe no regularity at all. One approach to palliate the effect of missing data on machine learning tasks is to replace the missing observations. Imputation algorithms attempt to calculate a value for a missing gap, using information associated with it, i.e., the attribute and/or other values in the same observation. While several imputation methods have been proposed in the literature, few works have addressed the question of the relationship between the type of missing data, the choice of the imputation method, and the effectiveness of classification algorithms that used the imputed data. In this paper we address the relationship among these three factors. By constructing a benchmark of hundreds of databases containing different types of missing data, and applying several imputation methods and classification algorithms, we empirically show that an interaction between imputation methods and supervised classification can be deduced. Besides, differences in terms of classification performance for the same imputation method in different missing data patterns have been found. This points to the convenience of considering the combined choice of the imputation method and the classifier algorithm according to the missing data type.

MISSING DATA IN LONGITUDINAL STUDIES OF AGING: THE GOOD, THE BAD, AND THE UGLY

In longitudinal studies of aging, missing data are inevitable. In the literature on the association between grip strength and survival time in older persons, the methods used to deal with missing values are generally not even mentioned. The default in standard statistical software is listwise deletion - removing all observations with any missing data. Missing data have been classified into three types: missing completely at random (the good), missing at random (the bad) and missing not at random (the ugly). Depending on the type of missing data, listwise deletion may produce biased results.The objectives of this presentation are: 1) To identify different types of missing data that may arise within a longitudinal study of aging and 2) To assess the estimates of the association between grip strength and mortality that result from applying different methods for handling missing data.Using data from the Cardiovascular Health Study, a longitudinal study of 5,201 older persons, we: 1) examined the types of missing data; 2) investigated the association between grip strength and survival time, applying three common methods of handling missing data: listwise deletion, last value carried forward (LOCF), and multiple imputation (MI). We identified all three types of missing data in the CHS. Compared to the method of listwise deletion, estimates of the association between grip strength and survival were weaker when applying LOCF, and stronger with MI. Methods of handling missing data in longitudinal studies of aging should be considered carefully in order to limit the potential for biased results.

Building Cross-National, Longitudinal Data Sets: Issues and Strategies for Implementation

With recent advances in computing power enhancing the ability of social scientists to analyze newly available “big data,” there is a new set of challenges for researchers building cross-national, longitudinal data sets. While scholars should embrace these new sources of data, we must carefully consider cleaning and coding decisions and how these influence our ultimate findings. In this article, we outline five common issues that researchers may face in building large cross-national, longitudinal data sets and suggest strategies for how to address each: (1) country data consistency including births, deaths, splits, unifications, and name changes, (2) longitudinal string matching, (3) identifying different types of missing data, (4) using these types of missing data in developing a theoretically and empirically grounded imputation strategy, and (5) understanding whether systemic change is driven by real world processes or by coding/cleaning choices. We also touch briefly on some general technical and technological considerations when working with large data sets. Throughout, we illustrate issues and strategies with examples drawn from our experience building a cross-national, longitudinal network data set of country-international nongovernmental organization memberships.

Data Missing Mechanism and Missing Data Real-Time Processing Methods in the Construction Monitoring of Steel Structures

The data link of construction monitoring of steel structures is usually long, complicated and with huge amount of data. Because of the influences of various factors during construction process, sometimes it is inevitable to cause partial monitoring data missing or interrupt of the data link which may affect the real-time control of construction process, even the structural safety assessment during construction. Based on the generating mechanism and characteristics of the missing data discovered from the statistic of actual construction monitoring of steel structures, this paper classifies the missing data into three types for developing new data processing methods. According to the characteristics of the different types of missing data, several compensating methods of the missing data are proposed respectively in this paper by regression analysis based on the principle of least square. The proposed methods are adopted to process the missing monitoring data of the steel structure construction of Art Gallery of Datong and the Circle of Life of Shenyang for verification. The estimated values are compared with the measured data. The numerical results show that the proposed missing data processing methods are effective and practical.

Local Feature Extraction Models from Incomplete Data in Face Recognition Based on Nonnegative Matrix Factorization

Data missing usually happens in the process of data collection, transmission, processing, preservation and application due to various reasons. In the research of face recognition, the missing of image pixel value will affect feature extraction. How to extract local feature from the incomplete data is an interesting as well as important problem. Nonnegative matrix factorization (NMF) is a low rank factorization method for matrix and has been successfully used in local feature extraction in various disciplines which face recognition is included. This paper mainly deals with this problem. Firstly, we classify the patterns of image pixel value missing, secondly, we provide the local feature extraction models basing on nonnegative matrix factorization under different types of missing data, thirdly, we compare the local feature extraction capabilities of the above given models under different missing ratio of the original data. Recognition rate is investigated under different data missing pattern. Numerical experiments are presented and conclusions are drawn at the end of the paper.

Ontological Considerations When Modeling Missing Data With Relational Databases

SQL’s use of nulls to indicate missing data in relational databases has been criticized as violating the relational model. In this article, I review this critique and two popular recommendations of resolution. The first is to retire SQL’s multivalued logic and replace it with a conventional binary logic. The second is to revise SQL’s implementation of multivalued logic so that it can recognize tautological propositions. I argue that underlying this debate is an ontological disagreement about how to properly model missing data and, more generally, social reality. I demonstrate that the relational model provides useful tools for modeling different types of missing data in different ways and, furthermore, offers a useful foundation upon which to conduct social research, one that supports both variable-oriented and case-oriented analysis.

How to avoid missing data and the problems they pose: design considerations.

One of the most common challenges in biomedical and psychosocial research is missing data, which occurs when respondents refuse to provide answers to sensitive questions and when study subjects are lost to follow-up during the repeated assessments of longitudinal trials. This paper is the first in a 3-part series focusing on this important topic; it describes different types of missing data and their differential effects on model estimates, focusing on study design strategies that can be used to prevent or minimize missing data and, thus, maintain the scientific integrity of the research. The second paper in the series will discuss implementation strategies to manage and reduce missing data while conducting the study, and the third paper will discuss analytic strategies for dealing with missing data after completion of data collection. It is always worth devoting careful attention to issues like missing data that may have significant effects on model estimates. As the saying goes, ‘an ounce of prevention is worth a pound of cure,’ so it is much better to focus on these issues during the planning stage of a study rather than having to deal with them later in the study. In this paper, we focus squarely on such preventive strategies as the first line of defense against the ubiquitous problem of missing data in clinical research studies. 1. Types of missing data and their effects on model estimates The reasons for missing data vary, and the degree to which missing data decreases the validity of the estimates depends on how the missing data arises. Thus it is important to make plausible assumptions about how missing data occurs in a study and, based on these assumptions, select appropriate models for addressing the effect of the missing data on inference from the observed results of the study. There are three statistical models with increasing levels of generalizability that are commonly used to classify different types of missing data. If missing data occur in a random fashion—that is, with no particular pattern that determines which data are observed and which are missing—it is typically referred to as ‘missing completely at random’ (MCAR). Data that are MCAR have no influence on any of the study participants’ outcomes and, thus, may be ignored because they do not result in an inferential bias. In many follow-up studies participants may be lost to follow up because of deteriorated or improved health conditions. In this situation the missed data are not MCAR because the probability of the missed visit depends on the outcome. For example, if an investigational medication worsens depression, subjects may drop out from the study over time, creating a so-called ‘monotone missing data’ pattern. In cases like this where whether or not data is missing is influenced by treatment-related effects, ignoring the missing data and focusing only on the subjects with complete data will usually give rise to biased estimates of treatment effects. If depression measures taken at baseline (or prior to dropping out) can be used to model this dependence structure in the missing data pattern (i.e., the relationship between baseline severity of depression and dropping out) this information can then be incorporated into the model for treatment effects to address the bias. If the dependence structure between the outcome of interest and the missing data can be modeled based on observed data, the missing data are classified as ‘missing at random’ (MAR).

Handling missing data issues in clinical trials for rheumatic diseases

Missing data are ubiquitous in clinical trials for rheumatic diseases, and it is important to accommodate them using appropriate statistical techniques. We review some of the basic considerations for missing data and survey a range of statistical techniques for analysis of longitudinal clinical trial data with missingness. Using clinical trial data from patients with diffuse systemic sclerosis, we show that different approaches to handling missing data can lead to different conclusions on the efficacy of the treatment. We then suggest how such discrepancies might be addressed. In particular, we emphasize that the commonly used method in rheumatic clinical trials of carrying the last observation forward to impute missing values should not be the primary analysis. We review software for analyzing different types of missing data and discuss our freely available software library for analyzing the more difficult but more realistic situation when the probability of dropout or missing data may depend on the unobserved missing value.

Measuring authoritative teaching

High quality measurements are important to evaluate interventions. The study reports on the development of a measurement to investigate authoritative teaching understood as a two-dimensional construct of warmth and control. Through the application of confirmatory factor analysis (CFA) and structural equation modelling (SEM) the factor structure and measurement invariance is investigated. Generally, results suggest that the two-dimensional model of authoritative teaching has satisfactory psychometric properties for longitudinal measurement invariance, ensuring the measurement of the same concept over time. Different types of missing data in this study are discussed. Also, the relevance of such study for professional development is addressed.

Impact of missing data on type 1 error rates in non‐inferiority trials

In this paper, a simulation study is conducted to systematically investigate the impact of different types of missing data on six different statistical analyses: four different likelihood-based linear mixed effects models and analysis of covariance (ANCOVA) using two different data sets, in non-inferiority trial settings for the analysis of longitudinal continuous data. ANCOVA is valid when the missing data are completely at random. Likelihood-based linear mixed effects model approaches are valid when the missing data are at random. Pattern-mixture model (PMM) was developed to incorporate non-random missing mechanism. Our simulations suggest that two linear mixed effects models using unstructured covariance matrix for within-subject correlation with no random effects or first-order autoregressive covariance matrix for within-subject correlation with random coefficient effects provide well control of type 1 error (T1E) rate when the missing data are completely at random or at random. ANCOVA using last observation carried forward imputed data set is the worst method in terms of bias and T1E rate. PMM does not show much improvement on controlling T1E rate compared with other linear mixed effects models when the missing data are not at random but is markedly inferior when the missing data are at random.

A pre-processing method to deal with missing values by integrating clustering and regression techniques

Data pre-processing is a critical task in the knowledge discovery process in order to ensure the quality of the data to be analyzed. One widely studied problem in data pre-processing is the handling of missing values with the aim to recover its original value. Based on numerous studies on missing values, it is shown that different methods are needed for different types of missing data. In this work, we propose a new method to deal with missing values in data sets where cluster properties exist among the data records. By integrating the clustering and regression techniques, the proposed method can predict the missing values with higher accuracy. To our best knowledge, this is the first work combining regression and clustering analysis to deal with the missing values problem. Through empirical evaluation, the proposed method was shown to perform better than other methods under different types of data sets.