Missing Data Situations Research Articles

Generative interpolation via a diffusion probabilistic model

Seismic data interpolation is essential in a seismic data processing workflow, recovering data from sparse sampling. Traditional and deep-learning-based methods have been widely used in the seismic data interpolation field and have achieved remarkable results. In this paper, we develop a seismic data interpolation method through the novel application of diffusion probabilistic models (DPMs). DPM transforms the complex end-to-end mapping problem into a progressive denoising problem, enhancing the ability to reconstruct complex situations of missing data, such as large proportions and large-gap missing data. The interpolation process begins with a standard Gaussian distribution and seismic data with missing traces and then removes noise iteratively with a UNet trained for different noise levels. Our DPM-based interpolation method allows interpolation for various missing cases, including regularly missing, irregularly missing, consecutively missing, noisy missing, and different ratios of missing cases. The ability to generalize to different seismic data sets is also discussed in this paper. Numerical results of synthetic and field data indicate satisfactory interpolation performance of the DPM-based interpolation method in comparison with the f- x prediction filtering method, the curvelet transform method, the low-dimensional manifold method (LDMM), and the coordinate attention-based UNet method, particularly in cases with large proportions and large-gap missing data. Diffusion is all we need for seismic data interpolation.

A Smart City Air Quality Data Imputation Method Using Markov Weights-Based Fuzzy Transfer Learning

An intelligent air quality monitoring system (IAQMS) is one of the key aspects of any smart city. The success of these monitoring systems largely depends on the quality of data. Missing air quality data is one of the crucial issues in any IAQMS, especially in small cities where sufficient historical air quality data is not available. In this study, a fuzzy transfer learning-based imputation (FTLI) method is proposed for the smart imputation of missing air quality data. The central concept of this proposed method is to acquire knowledge through fuzzy inference systems from other air quality monitoring systems called source domains where sufficient data is available. Later that knowledge is applied to impute the missing values of the target IAQMS (target domain) through some knowledge adaptation techniques. In this proposed method, Markov weights (transition probability matrix) are used to impute the missing values more accurately. The proposed imputation method is tested on the various missing data situations, which include random missing values, continuous missing values, and high missing rates. In this study, the performance of the proposed method is compared with other well-established methods, and the comparison results exhibit that the FTLI method outruns other methods under different missing rates.

Two-Step Approach to Processing Raw Strain Monitoring Data for Damage Detection of Structures under Operational Conditions.

Strain data of structural health monitoring is a prospective to be made full use of, because it reflects the stress peak and fatigue, especially sensitive to local stress redistribution, which is the probably damage in the vicinity of the sensor. For decoupling structural damage and masking effects caused by operational conditions to eliminate the adverse impacts on strain-based damage detection, small time-scale structural events, i.e., the short-term dynamic strain responses, are analyzed in this paper by employing unsupervised modeling. A two-step approach to successively processing the raw strain monitoring data in the sliding time window is presented, consisting of the wavelet-based initial feature extraction step and the decoupling step to draw damage indicators. The principal component analysis and a low-rank property-based subspace projection method are adopted as two alternative decoupling methodologies. The approach’s feasibility and robustness are substantiated by analyzing the strain monitoring data from a customized truss experiment to successfully remove the masking effects of operating loads and identify local damages even concerning accommodating situations of missing data and limited measuring points. This work also sheds light on the merit of a low-rank property to separate structural damages from masking effects by comparing the performances of the two optional decoupling methods of the distinct rationales.

Causal Algebras on Chain Event Graphs with Informed Missingness for System Failure.

Graph-based causal inference has recently been successfully applied to explore system reliability and to predict failures in order to improve systems. One popular causal analysis following Pearl and Spirtes et al. to study causal relationships embedded in a system is to use a Bayesian network (BN). However, certain causal constructions that are particularly pertinent to the study of reliability are difficult to express fully through a BN. Our recent work demonstrated the flexibility of using a Chain Event Graph (CEG) instead to capture causal reasoning embedded within engineers’ reports. We demonstrated that an event tree rather than a BN could provide an alternative framework that could capture most of the causal concepts needed within this domain. In particular, a causal calculus for a specific type of intervention, called a remedial intervention, was devised on this tree-like graph. In this paper, we extend the use of this framework to show that not only remedial maintenance interventions but also interventions associated with routine maintenance can be well-defined using this alternative class of graphical model. We also show that the complexity in making inference about the potential relationships between causes and failures in a missing data situation in the domain of system reliability can be elegantly addressed using this new methodology. Causal modelling using a CEG is illustrated through examples drawn from the study of reliability of an energy distribution network.

A Hierarchical Bayes Analysis and Comparison of PH Weibull and PH Exponential Models for One-Shot Device Testing Experiment

The one-shot devices are highly reliable and, therefore, accelerated life tests are often employed to perform the experiments on such devices. Obviously, in the process, some covariates are introduced. This paper considers the proportional hazards model to observe the effect of covariates on the failure rates under the assumption of two commonly used models, namely the exponential and the Weibull for the lifetimes. The Bayes implementation is proposed using the hybridization of Gibbs and Metropolis algorithms that routinely extend to missing data situations as well. The entertained models are compared using the Bayesian and deviance information criteria and the expected posterior predictive loss criterion. Finally, the results based on two real data examples are given as an illustration.

MISSING DATA SAMPLES: SYSTEMATIZATION AND CONDUCTING METHODS-A REVIEW

This paper investigates the phenomenon of the incomplete data samples by analyzing their structure and also resolves the necessary procedures regularly used in missing data analysis. The research gives a crucial perceptive of the techniques and mechanisms needed in dealing with missing data issues in general. The motivation for writing this brief overview of the topic lies in the fact that statistical researchers inevitably meet missing data in their analysis. The authors examine the applicability of regular approaches for handling the missing data situations. Based on several previously published results, the authors provide an example of the incomplete data sample model that can be implemented when confronting with specific missing data patterns.

Identification of a suitable technique for imputation of incomplete genotyping by sequencing (GBS) data

In the field of DNA sequencing, Genotype by sequencing is to discover SNPs in order to perform Genotyping studies. A most commonly occurring problem in GBS is the presence of missing observations. Quite often, the standard statistical models may not handle such missing data situation also known as incomplete data situations. An alternative to deal with incomplete data situation is to impute missing data for further downstream analysis. Hence a study is conducted with the objectives (i) to impute missing GBS data by various imputation techniques, based on both supervised and unsupervised learning algorithms at different levels of missingness, (ii) to identify suitable imputation technique to deal with incomplete GBS data situation. Based on correlation coefficient and mean squared prediction error (MSPE) between imputed value and true response, the accuracy of imputation technique was assessed. Different imputation techniques, viz., Mean Allele Frequency Imputation (MNI), Singular Value Decomposition Imputation (SVDI), k-Nearest Neighbour Imputation (kNNI), locally weighted linear regression imputation (LWI), Expectation Maximization Imputation (EMI) and Random Forest Imputation (RFI) were applied on incomplete GBS data of mice, a model animal organism, to assess their performance. The results revealed that RFI was found to be most accurate imputation technique. Besides, the performance of RFI in terms correlation coefficient at 5%, 10%, 15% and 20% missing data situation was observed to be 0.778, 0.765, 0.750 and 0.735 respectively. A Similar trend was also observed for RFI in terms of mean square prediction errors. Thus, it is suggested to use RFI technique to deal with incomplete GBS data situation and prior to the application of genomic selection models for breeding value estimation.

A bi-directional missing data imputation scheme based on LSTM and transfer learning for building energy data

Improving the energy efficiency of the buildings is a worldwide hot topic nowadays. To assist comprehensive analysis and smart management, high-quality historical data records of the energy consumption is one of the key bases. However, the energy data records in the real world always contain different kinds of problems. The most common problem is missing data. It is also one of the most frequently reported data quality problems in big data/machine learning/deep learning related literature in energy management. However, limited studied have been conducted to comprehensively discuss different kinds of missing data situations, including random missing, continuous missing, and large proportionally missing. Also, the methods used in previous literature often rely on linear statistical methods or traditional machine learning methods. Limited study has explored the feasibility of advanced deep learning and transfer learning techniques in this problem. To this end, this study proposed a methodology, namely the hybrid Long Short Term Memory model with Bi-directional Imputation and Transfer Learning (LSTM-BIT). It integrates the powerful modeling ability of deep learning networks and flexible transferability of transfer learning. A case study on the electric consumption data of a campus lab building was utilized to test the method. Results show that LSTM-BIT outperforms other methods with 4.24% to 47.15% lower RMSE under different missing rates.

Bayesian inference for Common cause failure rate based on causal inference with missing data

This paper proposes a methodology to handle the causality to make inference on common cause failure (CCF) in a missing data situation. The data are collected in the form of contingency tables but the only available tokens of information are the numbers of CCFs of different orders and the numbers of failures due to a given cause, i.e. the margins of the contingency table. The frequencies in each cell are unknown; we are in a situation of missing data. Assuming a Poisson model for the count, we suggest a Bayesian approach and we use the inverse Bayes formula (IBF) combined with a Metropolis-Hastings algorithm to make inference on the parameters. The performance of the resulting algorithm is evaluated through simulations. A comparison is made by analogy with results obtained from the recently proposed α-composition method.

Supervised Variational Autoencoders for Soft Sensor Modeling With Missing Data

Autoencoder (AE) is a deep neural network that has been widely utilized in process industry owing to its superior abilities of feature extraction and data reconstruction. Recently, assuming the latent variables to be random variables, a probabilistic variant of it called variational autoencoder (VAE) has achieved a major success in different applications. In this article, we develop two novel submodels based on deep VAEs (DVAE), which are further utilized to establish a soft sensor framework. By the use of our first submodel known as supervised DVAE (SDVAE), the distribution information of latent features can be obtained. This is used as a prior of the second submodel known as the modified unsupervised DVAE (MUDVAE). Then, a new soft sensor framework can be constructed by combing the encoder of SDVAE with the decoder of MUDVAE. Since our designed VAE has superior ability in data reconstruction, it also works well under the missing data situation which is common in process industries due to sensor failures. Thus, we extend the proposed soft sensor framework to handle the missing data situation. The effectiveness of our proposed soft sensor frameworks is finally demonstrated via an industrial polymerization dataset.

Model-based co-clustering for mixed type data

The importance of clustering for creating groups of observations is well known. The emergence of high-dimensional data sets with a huge number of features leads to co-clustering techniques, and several methods have been developed for simultaneously producing groups of observations and features. By grouping the data set into blocks (the crossing of a row-cluster and a column-cluster), these techniques can sometimes better summarize the data set and its inherent structure. The Latent Block Model (LBM) is a well-known method for performing co-clustering. However, recently, contexts with features of different types (here called mixed type data sets) are becoming more common. The LBM is not directly applicable to this kind of data set. Here a natural extension of the usual LBM to the “Multiple Latent Block Model” (MLBM) is proposed in order to handle mixed type data sets. Inference is performed using a Stochastic EM-algorithm that embeds a Gibbs sampler, and allows for missing data situations. A model selection criterion is defined to choose the number of row and column clusters. The method is then applied to both simulated and real data sets.

Accounting for missing data in statistical analyses: multiple imputation is not always the answer.

BackgroundMissing data are unavoidable in epidemiological research, potentially leading to bias and loss of precision. Multiple imputation (MI) is widely advocated as an improvement over complete case analysis (CCA). However, contrary to widespread belief, CCA is preferable to MI in some situations.MethodsWe provide guidance on choice of analysis when data are incomplete. Using causal diagrams to depict missingness mechanisms, we describe when CCA will not be biased by missing data and compare MI and CCA, with respect to bias and efficiency, in a range of missing data situations. We illustrate selection of an appropriate method in practice.ResultsFor most regression models, CCA gives unbiased results when the chance of being a complete case does not depend on the outcome after taking the covariates into consideration, which includes situations where data are missing not at random. Consequently, there are situations in which CCA analyses are unbiased while MI analyses, assuming missing at random (MAR), are biased. By contrast MI, unlike CCA, is valid for all MAR situations and has the potential to use information contained in the incomplete cases and auxiliary variables to reduce bias and/or improve precision. For this reason, MI was preferred over CCA in our real data example.ConclusionsChoice of method for dealing with missing data is crucial for validity of conclusions, and should be based on careful consideration of the reasons for the missing data, missing data patterns and the availability of auxiliary information.

A regression approach to vowel normalization for missing and unbalanced data.

Researchers investigating the vowel systems of languages or dialects frequently employ normalization methods to minimize between-speaker variability in formant patterns while preserving between-phoneme separation and (socio-)dialectal variation. Here two methods are considered: log-mean and Lobanov normalization. Although both of these methods express formants in a speaker-dependent space, the methods differ in their complexity and in their implied models of human vowel-perception. Typical implementations of these methods rely on balanced data across speakers so that researchers may have to reduce the data available in the analyses in missing-data situations. Here, an alternative method is proposed for the normalization of vowels using the log-mean method in a linear-regression framework. The performance of the traditional approaches to log-mean and Lobanov normalization against the regression approach to the log-mean method using naturalistic, simulated vowel-data was investigated. The results indicate that the Lobanov method likely removes legitimate linguistic variation from vowel data and often provides very noisy estimates of the actual vowel quality associated with individual tokens. The authors further argue that the Lobanov method is too complex to represent a plausible model of human vowel perception, and so is unlikely to provide results that reflect the true perceptual organization of linguistic data.

Enhancing the applicability of Kohonen Self-Organizing Map (KSOM) estimator for gap-filling in hydrometeorological timeseries data

The Kohonen Self-Organizing Map (KSOM) estimator is prescribed as a useful tool for infilling the missing data in hydrometeorology. However, in this study, when the performance of the KSOM estimator is tested for gap-filling in the streamflow, rainfall, evapotranspiration (ET), and temperature timeseries data, collected from 30 gauging stations in India under missing data situations, it is felt that the KSOM modeling performance could be further improved. Consequently, this study tries to answer the research questions as to whether the length of record of the historical data and its variability has any effect on the performance of the KSOM? Whether inclusion of temporal distribution of timeseries data and the nature of outliers in the KSOM framework enhances its performance further? Subsequently, it is established that the KSOM framework should include the coefficient of variation of the datasets for determination of the number of map units, without considering it as a single value function of the sample data size. This could help to upscale and generalize the applicability of KSOM for varied hydrometeorological data types.

Inference for a Failure Counting Process Partially Observed

We consider a system defined as a collection of two types of components. The number of failures of each component is described as a stochastic process, with one of the processes depending on the other. None of the processes is observed directly. The only available information is the number of type 1 components at risk in the system. Because of this missing data situation, different algorithms relying on an Expectation Maximization (EM) strategy are proposed to obtain the MLE of the intensity parameters for both processes so we can assess the reliability of type 1 and type 2 components. To overcome the computational limits of EM, a Monte Carlo EM (MCEM) algorithm using a Metropolis procedure is presented. Stochastic EM (SEM) algorithms including a Bayesian approach are also described. The methods are applied to simulated data to demonstrate their efficiency.

Mixtures of spatial spline regressions for clustering and classification

Classification and clustering of functional data arise in many areas of modern research. Currently, techniques for performing such tasks have concentrated on applications to univariate functions. Such techniques can be extended to the domain of classifying and clustering bivariate functions (i.e. surfaces) over rectangular domains. This is achieved by combining the current techniques in spatial spline regression (SSR) with finite mixture models and mixed-effects models. As a result, three novel techniques have been developed: spatial spline mixed models (SSMM) for fitting populations of surfaces, mixtures of SSR (MSSR) for clustering surfaces, and MSSR discriminant analysis (MSSRDA) for classification of surfaces. Through simulations and applications to problems in handwritten character recognition, it is shown that SSMM, MSSR, and MSSRDA are effective in performing their desired tasks. It is also shown that in the context of handwritten character recognition, MSSR and MSSRDA are comparable to established methods, and are able to outperform competing approaches in missing-data situations.

Missing data in a multi-item instrument were best handled by multiple imputation at the item score level

ObjectivesRegardless of the proportion of missing values, complete-case analysis is most frequently applied, although advanced techniques such as multiple imputation (MI) are available. The objective of this study was to explore the performance of simple and more advanced methods for handling missing data in cases when some, many, or all item scores are missing in a multi-item instrument. Study Design and SettingReal-life missing data situations were simulated in a multi-item variable used as a covariate in a linear regression model. Various missing data mechanisms were simulated with an increasing percentage of missing data. Subsequently, several techniques to handle missing data were applied to decide on the most optimal technique for each scenario. Fitted regression coefficients were compared using the bias and coverage as performance parameters. ResultsMean imputation caused biased estimates in every missing data scenario when data are missing for more than 10% of the subjects. Furthermore, when a large percentage of subjects had missing items (>25%), MI methods applied to the items outperformed methods applied to the total score. ConclusionWe recommend applying MI to the item scores to get the most accurate regression model estimates. Moreover, we advise not to use any form of mean imputation to handle missing data.

Statistical Analysis of Noncommensurate Multiple Outcomes

Many studies collect multiple outcomes to characterize treatment effectiveness or evaluate risk factors. These outcomes tend to be correlated because they are measuring related quantities in the same individuals, but the common approach used by researchers is to ignore this correlation and analyze each outcome separately. There may be advantages to consider the simultaneous analysis of the outcomes using multivariate methods. Although the joint analysis of outcomes measured in the same scale (commensurate outcomes) can be undertaken with standard statistical methods, outcomes measured in different scales (noncommensurate outcomes), such as mixed binary and continuous outcomes, present more difficult challenges. In this article, we contrast some statistical approaches to analyze noncommensurate multiple outcomes. We discuss the advantages of a multivariate method for the analysis of noncommensurate outcomes, including situations of missing data. A real data example from a clinical trial, comparing bare-metal with sirolimus-eluting stents, is used to illustrate the differences between the statistical approaches.

Statistical Approaches to Modeling: Multiple Outcomes in Psychiatric Studies

Increasingly, multiple outcomes are collected in order to characterize treatment effectiveness or to evaluate risk factors. These outcomes tend to be correlated because they are measuring related quantities in the same individuals. While the analysis of outcomes measured in the same scale (commensurate outcomes) can be undertaken with standard statistical methods, outcomes measured in different scales (non-commensurate outcomes), such as mixed binary and continuous outcomes, present more difficult challenges.In this paper we contrast some statistical approaches to analyze non-commensurate multiple outcomes. We discuss the advantages of a multivariate method for the analysis of non-commensurate outcomes including situations of missing data. A real data example from a clinical trial, comparing different treatments for depression in low-income women, is used to illustrate the differences between the statistical approaches.

Testing intergroup concordance in ranking experiments with two groups of judges.

Across many areas of psychology, concordance is commonly used to measure the (intragroup) agreement in ranking a number of items by a group of judges. Sometimes, however, the judges come from multiple groups, and in those situations, the interest is to measure the concordance between groups, under the assumption that there is some within-group concordance. In this investigation, existing methods are compared under a variety of scenarios. Permutation theory is used to calculate the error rates and the power of the methods. Missing data situations are also studied. The results indicate that the performance of the methods depend on (a) the number of items to be ranked, (b) the level of within-group agreement, and (c) the level of between-group agreement. Overall, using the actual ranks of the items gives better results than using the pairwise comparison of rankings. Missing data lead to loss in statistical power, and in some cases, the loss is substantial. The degree of power loss depends on the missing mechanism and the method of imputing the missing data, among other factors.