Data Imputation Approaches Research Articles

Low-Rank Tensor and Hybrid Smoothness Regularization-Based Approach for Traffic Data Imputation With Multimodal Missing

Low-Rank Tensor and Hybrid Smoothness Regularization-Based Approach for Traffic Data Imputation With Multimodal Missing

ScDTL: enhancing single-cell RNA-seq imputation through deep transfer learning with bulk cell information.

The increasing single-cell RNA sequencing (scRNA-seq) data enable researchers to explore cellular heterogeneity and gene expression profiles, offering a high-resolution view of the transcriptome at the single-cell level. However, the dropout events, which are often present in scRNA-seq data, remaining challenges for downstream analysis. Although a number of studies have been developed to recover single-cell expression profiles, their performance may be hindered due to not fully exploring the inherent relations between genes. To address the issue, we propose scDTL, a deep transfer learning based approach for scRNA-seq data imputation by harnessing the bulk RNA-sequencing information. We firstly employ a denoising autoencoder trained on bulk RNA-seq data as the initial imputation model, and then leverage a domain adaptation framework that transfers the knowledge learned by the bulk imputation model to scRNA-seq learning task. In addition, scDTL employs a parallel operation with a 1D U-Net denoising model to provide gene representations of varying granularity, capturing both coarse and fine features of the scRNA-seq data. Finally, we utilize a cross-channel attention mechanism to fuse the features learned from the transferred bulk imputation model and U-Net model. In the evaluation, we conduct extensive experiments to demonstrate that scDTL could outperform other state-of-the-art methods in the quantitative comparison and downstream analyses.

Autoencoder imputation of missing heterogeneous data for Alzheimer's disease classification

AbstractMissing Alzheimer's disease (AD) data is prevalent and poses significant challenges for AD diagnosis. Previous studies have explored various data imputation approaches on AD data, but the systematic evaluation of deep learning algorithms for imputing heterogeneous and comprehensive AD data is limited. This study investigates the efficacy of denoising autoencoder‐based imputation of missing key features of heterogeneous data that comprised tau‐PET, MRI, cognitive and functional assessments, genotype, sociodemographic, and medical history. The authors focused on extreme (≥40%) missing at random of key features which depend on AD progression; identified as the history of a mother having AD, APoE ε4 alleles, and clinical dementia rating. Along with features selected using traditional feature selection methods, latent features extracted from the denoising autoencoder are incorporated for subsequent classification. Using random forest classification with 10‐fold cross‐validation, robust AD predictive performance of imputed datasets (accuracy: 79%–85%; precision: 71%–85%) across missingness levels, and high recall values with 40% missingness are found. Further, the feature‐selected dataset using feature selection methods, including autoencoder, demonstrated higher classification score than that of the original complete dataset. These results highlight the effectiveness and robustness of autoencoder in imputing crucial information for reliable AD prediction in AI‐based clinical decision support systems.

Collaborative Structure-Preserved Missing Data Imputation for Single-Cell RNA-Seq Clustering.

Clustering of the single-cell RNA-seq (scRNA-seq) transcriptome profiles is able to identify cell types, which is beneficial to improve the understanding of disease progression. However, in practice, the single-cell expression data often contains a significant number of missing values as a result of technical variability. Missing data is a critical challenge in scRNA-seq clustering analysis since the unknown value does not reflect the underlying true expression level and makes it difficult to discovering cell types by applying clustering algorithms directly. Various approaches have been developed to overcome missing data issue in scRNA-seq clustering. Most of them recover missing expression values by borrowing observed data from similar cells or synthesizing data via generative adversarial networks. Such that the biologically meaningful cluster structure has not been sufficiently exploited. In this work, we introduce ColImpute, a collaborative structure-preserved missing data imputation approach for the scRNA-seq clustering. Specifically, a cluster structure-preserved imputation module and a subspace clustering module, which respectively perform missing data imputation and cell subtypes identification, are integrated into a unified optimization framework to train the two networks in a collaborative manner. Consequently, the clustering module effectively contributes cluster-structure information to guide the trainning process of the missing data imputation module. Simultaneously, the cluster structure-preserved imputation module reciprocally enhances the performance of the clustering module by generating more precise recovered samples. Promising experimental results show that the proposed method is effective for both the data imputation and the cell types identification.

Comparing the Performance of Different Missing Data Imputation Approaches in Discrete Outcome Modeling

Comparing the Performance of Different Missing Data Imputation Approaches in Discrete Outcome Modeling

Evaluation of data imputation approaches for multi-stream building systems data1

Increasing advancements in building digitization, smart sensing, and metering technologies have allowed large amounts of timeseries data to be collected for monitoring, analyzing, and controlling building systems. However, due to sensor or communication failures, the data collected are often incomplete and poor in quality. Data imputation approaches to replace the missing values, specifically based on either statistical or predictive models have been widely adopted for multivariate datasets in other domains. It is hence of interest to find an effective way to impute timeseries data collected from a building system. In this paper, we evaluate multiple data imputation approaches using data collected from a medium sized building situated in Stockholm, Sweden and a small commercial building from the ASHRAE RP-1312 research project. Sensors with widely varying characteristics from the case study buildings were selected to evaluate the imputation methods. The imputation accuracy and the impact of each chosen imputation method on information entropy, short-term building forecasting model performance, and fault detection strategy were evaluated. Results demonstrate that incorporating time-lagged cross correlations within a k -nearest neighbor ( k NN) model provide the most accurate imputations without affecting the quality of subsequent data analysis.

Artificial neural network-based data imputation for handling anomalous energy consumption readings in smart homes

Smart homes are at the forefront of sustainable living, utilizing advanced monitoring systems to optimize energy consumption. However, these systems frequently encounter issues with anomalous data such as missing data, redundant data, and outliers data which can undermine their effectiveness. In this paper, an artificial neural network (ANN)-based approach for data imputation is specifically designed to deal with the anomalies in smart home energy consumption datasets. Our research harnesses the power of ANNs to model intricate patterns within energy consumption data, enabling the accurate imputation of missing values while detecting and rectifying anomalous data. This approach not only enhances the completeness of the data but also augments its overall quality, ensuring more reliable results. To evaluate the effectiveness of our ANN-based imputation method, comprehensive experiments were conducted using real-world smart home energy consumption datasets. Our findings demonstrate that this approach outperforms traditional imputation techniques like mean imputation and median imputation in terms of accuracy. Furthermore, it showcases adaptability to diverse smart home scenarios and datasets, making it a versatile solution for improving data quality. In conclusion, this study introduces an advanced data imputation technique based on ANNs, tailor-made for addressing anomalies in smart home energy consumption data. Beyond merely filling data gaps, this approach elevates the dataset's reliability and completeness, thereby facilitating a more precise analysis of energy consumption and supporting informed decision-making in the context of smart homes and sustainable energy management. Ultimately, the proposed method has the potential to contribute considerably to the ongoing evolution of smart home technologies and energy conservation efforts.

Erratum to: A visual analysis approach for data imputation via multi-party tabular data correlation strategies

Erratum to: A visual analysis approach for data imputation via multi-party tabular data correlation strategies

An evolutionary computation classification method for high‐dimensional mixed missing variables data

AbstractData missing is a prevalent issue in various real‐world systems. It may deteriorate the performance of classification algorithms running on these platforms. Numerous effective imputation methods exist to address this problem. However, traditional data imputation approaches mainly focus on low‐dimensional missing data, and in addition, they do not make use of the randomness of the missing values and the information of labels simultaneously. To solve these problems, the authors propose a novel data imputation algorithm, named Particle Swarm Optimization for High‐dimensional mixed Missing variables data (PSOHM). PSOHM introduces a feature filtering algorithm for feature selection on missing data, followed by a feature discrimination method to categorize chosen features. PSOHM then employs particle swarm optimization to optimize imputation functions for both continuous and discrete features. Continuous features are modelled as Gaussian distributions, with the mean and standard deviation encoded into particles. Additionally, the probabilities of values for discrete features are also encoded. Moreover, accuracy serves as the optimization objective, utilizing both the randomness of missing values and the label information to improve the algorithm's performance. Six typical algorithms are employed to make a comparison. The results demonstrate that the authors’ method is superior to the compared approaches on the six different kinds of classical datasets.

A novel missing data imputation approach based on clinical conditional Generative Adversarial Networks applied to EHR datasets

The missing data mechanism is a relevant problem in Machine Learning (ML) and biomedical informatics communities. Real-world Electronic Health Record (EHR) datasets comprise several missing values, thus revealing a high level of spatiotemporal sparsity in the predictors’ matrix. Several approaches in the state-of-the-art tried to deal with this problem by proposing different data imputation strategies that (i) are often unrelated to the ML model, (ii) are not conceived for EHR data where laboratory exams are not prescribed uniformly over time and percentage of missing values is high (iii) exploit only univariate and linear information on the observed features. Our paper proposes a data imputation strategy based on a clinical conditional Generative Adversarial Network (ccGAN) capable of imputing missing values by exploiting non-linear and multivariate information across patients. Unlike other GAN data imputation-based approaches, our method deals explicitly with the high level of missingness of routine EHR data by conditioning the imputing strategy to the observable values and those fully-annotated. We demonstrated the statistical significance of the ccGAN to other state-of-the-art approaches in terms of imputation (around 19.79% of gain to the best competitor) and predictive performance (up to 1.60% of gain to the best competitor) on a real multi-diabetic centers dataset. We also demonstrated its robustness across different missingness rates (up to 1.61% of gain to the best competitor in the highest missingness rates condition) on an additional benchmark EHR dataset.

Impact of connected corridor volume data imputations on digital twin performance measures

To fully leverage “smart” transportation infrastructure data-stream investments, the creation of applications that provide real-time meaningful and actionable corridor-performance metrics is needed. However, the presence of gaps in data streams can lead to significant application implementation challenges. To demonstrate and help address these challenges, a digital twin smart-corridor application case study is presented with two primary research objectives: (1) explore the characteristics of volume data gaps on the case study corridor, and (2) investigate the feasibility of prioritizing data streams for data imputation to drive the real-time application. For the first objective, a K-means clustering analysis is used to identify similarities and differences among data gap patterns. The clustering analysis successfully identifies eight different data loss patterns. Patterns vary in both continuity and density of data gap occurrences, as well as time-dependent losses in several clusters. For the second objective, a temporal-neighboring interpolation approach for volume data imputation is explored. When investigating the use of temporal-neighboring interpolation imputations on the digital twin application, performance is, in part, dependent on the combination of intersection approaches experiencing data loss, demand relative to capacity at individual locations, and the location of the loss along the corridor. The results indicate that these insights could be used to prioritize intersection approaches suitable for data imputation and to identify locations that require a more sensitive imputation methodology or improved maintenance and monitoring.

Quantitative Evaluation of Imputation Methods Using Bounds Estimation of the Coefficient of Determination for Data-Driven Models with an Application to Drilling Logs

Summary With the constantly increasing quantity of data recorded in the oil and gas industry, data analytics and data-driven algorithms are gaining popularity. Meanwhile, they are highly sensitive to employed data management methods. Using drilling data as an example, the methods for data quality improvement play an ever increasing role in the data preparation phase. One of the most common data issues, especially in real-time sensing systems, is missing data (gaps between measured points). Various data imputation approaches (forward filling, interpolation, regression modeling, etc.) have been used to fill gaps to complete data sets as standard data processing procedures. The metrics, such as coefficient of determination (R2), mean absolute error, root mean square error, and mean absolute percentage error, are common ways to evaluate the imputation approaches, assuming that the ground truth data are at hand. In reality, the ground truth of missing data is not available for quantitative method evaluation. Especially when data are received occasionally, the ground truth is impossibly or inaccurately estimated, leading to tricky situations. For instance, how to evaluate infilling methods quantitatively or how to compare the methods’ behaviors? To the best of the authors’ knowledge, there are so far few existing methods to quantitatively estimate the accuracy of data imputation without the ground truth. To some extent, one may lack proven confidence in using imputed data for following up data analysis and data-driven modeling. In this study, a novel approach has been developed to quantitatively evaluate the data imputation quality. The presented method is built on an analytical estimation of the R2 bounds in the context of linear interpolation. The reasons for choosing linear regression as a benchmarking method are twofold. First, the bounds evaluation of R2 can be determined analytically for data imputation in the context of linear regression. Second, linear regression is a relatively simple imputation method, and we assume that more sophisticated methods, such as random forest (RF), k-nearest neighbors (KNN), and other iterative imputation techniques, can achieve higher accuracy with proper parameter tuning. The evaluation method is demonstrated by examples using real-time drilling data logs. Moreover, the application to drilling logs also enables informed decision-making in relation to gap filling and data resampling workflows.

Missing Value Imputation Methods for Electronic Health Records

Electronic health records (EHR) are patient-level information, e.g., laboratory tests and questionnaires, stored in electronic format. Compared to physical records, the EHR alternative allows patients to access their data easily and helps staff with management procedural tasks such as information sharing across different organizations. Moreover, this type of data is commonly used by researchers for predictive and classification purposes, employing statistical and machine learning methods. However, missingness is a phenomenon that is observed very frequently for such measurements. Even though this missingness is often significant, it is usually treated poorly with either case deletion or simple methods, resulting in suboptimal and/or inaccurate predictive results. This happens because the simple methods, e.g., k-nearest neighbors (kNN) and mean/mode imputation, fail in most cases to incorporate the complex relationships that define these medical datasets. To address these limitations, in this paper we test and improve state-of-the-art missing data imputation models and practices. We propose a new missing value imputation method based on denoising autoencoders (DAE) with kNN for the pre-imputation task. We optimize the training methodology by re-applying kNN to the missing data every <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> epochs using a different value for the variable <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> each time to yield more accurate results. We also revise a state-of-the-art missing data imputation approach based on a generative adversarial network (GAN). Using this as a baseline, we introduce improvements regarding both the architecture and the training procedure. These models are compared with the ones usually employed within clinical research studies for both the task of imputation and post-imputation prediction. Results show that our proposed deep learning approaches outperform the standard baselines, yielding better imputation and predictive results.

An Improved Bi-LSTM-Based Missing Value Imputation Approach for Pregnancy Examination Data

In recent years, the development of computer technology has promoted the informatization and intelligentization of hospital management systems and thus produced a large amount of medical data. These medical data are valuable resources for research. We can obtain inducers and unknown symptoms that can help discover diseases and make earlier diagnoses. Hypertensive disorder in pregnancy (HDP) is a common obstetric complication in pregnant women, which has severe adverse effects on the life safety of pregnant women and fetuses. However, the early and mid-term symptoms of HDP are not obvious, and there is no effective solution for it except for terminating the pregnancy. Therefore, detecting and preventing HDP is of great importance. This study aims at the preprocessing of pregnancy examination data, which serves as a part of HDP prediction. We found that the problem of missing data has a large impact on HDP prediction. Unlike general data, pregnancy examination data have high dimension and a high missing rate, are in a time series, and often have many non-linear relations. Current methods are not able to process the data effectively. To this end, we propose an improved bi-LSTM-based missing value imputation approach. It combines traditional machine learning and bidirectional LSTM to deal with missing data of pregnancy examination data. Our missing value imputation method obtains a good effect and improves the accuracy of the later prediction of HDP using examination data.

Missing data imputation using an iterative denoising autoencoder (IDAE) for dissolved gas analysis

• A data imputation method is proposed to address transformers with multiple missing DGA data. • An iterative imputation method is demonstrated, based on a denoising autoencoder. • Features are extracted from both gas trend and composition ratios for better imputation. • The proposed method is validated using a vast amount of field data. With the expansion of the energy market, safe and stable operation of the electrical power system has become an important issue. In an effort to achieve this goal, much research has been conducted on diagnosis approaches suitable for core components of the electrical power system. Transformers are one such core component. Most of the research on transformers has focused on developing a diagnosis model; less effort has been devoted to the data, in spite of the fact that such models require data of sufficient quantity and quality, which is not usually readily available for transformers. Thus, in this paper, we propose a way to fully exploit the valuable transformer data, using a data imputation approach called the iterative denoising autoencoder (IDAE) method. The proposed method imputes missing values of dissolved gas analysis (DGA) data, which is frequently lost, for various reasons. IDAE can help diagnose the health state of transformers accurately by estimating the missing values of DGA data. The proposed method is verified in this research through three comparative studies that examine field data provided by an electric power corporation. Specific studies provide: (1) a comparison with conventional methods on imputation performance for a single gas, (2) examination of imputation performance between multiple missing values, and (3) documentation of diagnosis accuracy before and after imputation. The results of the case studies show that the proposed method is effective for imputation of the missing DGA data.

AE-TPGG: a novel autoencoder-based approach for single-cell RNA-seq data imputation and dimensionality reduction.

Single-cell RNA sequencing (scRNA-seq) technology has become an effective tool for high-throughout transcriptomic study, which circumvents the averaging artifacts corresponding to bulk RNA-seq technology, yielding new perspectives on the cellular diversity of potential superficially homogeneous populations. Although various sequencing techniques have decreased the amplification bias and improved capture efficiency caused by the low amount of starting material, the technical noise and biological variation are inevitably introduced into experimental process, resulting in high dropout events, which greatly hinder the downstream analysis. Considering the bimodal expression pattern and the right-skewed characteristic existed in normalized scRNA-seq data, we propose a customized autoencoder based on a two-part-generalized-gamma distribution (AE-TPGG) for scRNA-seq data analysis, which takes mixed discrete-continuous random variables of scRNA-seq data into account using a two-part model and utilizes the generalized gamma (GG) distribution, for fitting the positive and right-skewed continuous data. The adopted autoencoder enables AE-TPGG to captures the inherent relationship between genes. In addition to the ability of achieving low-dimensional representation, the AE-TPGG model also provides a denoised imputation according to statistical characteristic of gene expression. Results on real datasets demonstrate that our proposed model is competitive to current imputation methods and ameliorates a diverse set of typical scRNA-seq data analyses.Electronic Supplementary MaterialSupplementary material is available in the online version of this article at 10.1007/s11704-022-2011-y.

SUMMIT: An integrative approach for better transcriptomic data imputation improves causal gene identification

Genes with moderate to low expression heritability may explain a large proportion of complex trait etiology, but such genes cannot be sufficiently captured in conventional transcriptome-wide association studies (TWASs), partly due to the relatively small available reference datasets for developing expression genetic prediction models to capture the moderate to low genetically regulated components of gene expression. Here, we introduce a method, the Summary-level Unified Method for Modeling Integrated Transcriptome (SUMMIT), to improve the expression prediction model accuracy and the power of TWAS by using a large expression quantitative trait loci (eQTL) summary-level dataset. We apply SUMMIT to the eQTL summary-level data provided by the eQTLGen consortium. Through simulation studies and analyses of genome-wide association study summary statistics for 24 complex traits, we show that SUMMIT improves the accuracy of expression prediction in blood, successfully builds expression prediction models for genes with low expression heritability, and achieves higher statistical power than several benchmark methods. Finally, we conduct a case study of COVID-19 severity with SUMMIT and identify 11 likely causal genes associated with COVID-19 severity.

Exploring Computational Data Amplification and Imputation for the Discovery of Type 1 Diabetes (T1D) Biomarkers from Limited Human Datasets.

Background: Type 1 diabetes (T1D) is a devastating disease with serious health complications. Early T1D biomarkers that could enable timely detection and prevention before the onset of clinical symptoms are paramount but currently unavailable. Despite their promise, omics approaches have so far failed to deliver such biomarkers, likely due to the fragmented nature of information obtained through the single omics approach. We recently demonstrated the utility of parallel multi-omics for the identification of T1D biomarker signatures. Our studies also identified challenges. Methods: Here, we evaluated a novel computational approach of data imputation and amplification as one way to overcome challenges associated with the relatively small number of subjects in these studies. Results: Using proprietary algorithms, we amplified our quadra-omics (proteomics, metabolomics, lipidomics, and transcriptomics) dataset from nine subjects a thousand-fold and analyzed the data using Ingenuity Pathway Analysis (IPA) software to assess the change in its analytical capabilities and biomarker prediction power in the amplified datasets compared to the original. These studies showed the ability to identify an increased number of T1D-relevant pathways and biomarkers in such computationally amplified datasets, especially, at imputation ratios close to the “golden ratio” of 38.2%:61.8%. Specifically, the Canonical Pathway and Diseases and Functions modules identified higher numbers of inflammatory pathways and functions relevant to autoimmune T1D, including novel ones not identified in the original data. The Biomarker Prediction module also predicted in the amplified data several unique biomarker candidates with direct links to T1D pathogenesis. Conclusions: These preliminary findings indicate that such large-scale data imputation and amplification approaches are useful in facilitating the discovery of candidate integrated biomarker signatures of T1D or other diseases by increasing the predictive range of existing data mining tools, especially when the size of the input data is inherently limited.

Explainable multisensor fusion-based automatic reconciliation and imputation of faulty and missing data in membrane bioreactor plants for fouling alleviation and energy saving

A membrane bioreactor (MBR) is a crucial wastewater treatment unit that requires continuous and precise monitoring to ensure stable operation and avoid energy loss. Sensor malfunction in MBR plants leads to missing and faulty measurements that negatively affect the control of membrane fouling and energy consumption. This study proposes an intelligent monitoring system to alleviate membrane fouling by the autonomous handling of sensor malfunctions in MBR plants using explainable AI (XAI) and a new multisensor fusion-based automated data reconciliation and imputation (MSF-ARI) approach. Missing and faulty data cases were used to validate the MSF-ARI approach in imputing the consecutive and point-to-point missing values and the detection, diagnosis, and reconciliation of the faulty data. Then, XAI analysis and an integrated biological-physical MBR model were utilized to evaluate the effect of MSF-ARI on membrane fouling and energy consumption. The results showed that the proposed MSF-ARI presents a superior missing data imputation performance (MAE = 0.31 mg/L) and a high reconciliation of the faulty measurements (MAE = 1.96 mg/L) along with a high detection rate (DRSPE = 100 %) and well-diagnosed fault groups. Applying MSF-ARI, the early membrane fouling was avoided by preventing the increased accumulation of the stable sludge cake, leading to a prolonged MBR operation by 10 days with an energy saving of 37.44 %. Therefore, managing faulty and missing data by MSF-ARI could contribute to the mitigation of membrane fouling and achieve a sustainable balance between operational time and energy consumption.

ScMTD: a statistical multidimensional imputation method for single-cell RNA-seq data leveraging transcriptome dynamic information

BackgroundSingle-cell RNA sequencing (scRNA-seq) provides a powerful tool to capture transcriptomes at single-cell resolution. However, dropout events distort the gene expression levels and underlying biological signals, misleading the downstream analysis of scRNA-seq data.ResultsWe develop a statistical model-based multidimensional imputation algorithm, scMTD, that identifies local cell neighbors and specific gene co-expression networks based on the pseudo-time of cells, leveraging information on cell-level, gene-level, and transcriptome dynamic to recover scRNA-seq data. Compared with the state-of-the-art imputation methods through several real-data-based analytical experiments, scMTD effectively recovers biological signals of transcriptomes and consistently outperforms the other algorithms in improving FISH validation, trajectory inference, differential expression analysis, clustering analysis, and identification of cell types.ConclusionsscMTD maintains the gene expression characteristics, enhances the clustering of cell subpopulations, assists the study of gene expression dynamics, contributes to the discovery of rare cell types, and applies to both UMI-based and non-UMI-based data. Overall, scMTD’s reliability, applicability, and scalability make it a promising imputation approach for scRNA-seq data.