Reconstruction Of Missing Information Research Articles

GLHDR: HDR video reconstruction driven by global to local alignment strategy

Reconstructing High Dynamic Range (HDR) video from alternating exposure Low Dynamic Range (LDR) sequence is an exceptionally challenging task. It not only demands the reliable reconstruction of missing information caused by occlusion or motion without introducing artifacts but also balances the exposure differences between frames to ensure a visually pleasing reconstructed HDR video. Unfortunately, existing methods are typically complex and struggle with unavoidable artifacts and noise, especially when dealing with low-exposed scenes. To tackle this formidable challenge, we propose a two-stage HDR video reconstruction method that employs a global to local alignment strategy. Firstly, we utilize iterative optical flow estimation and hybrid weighting to achieve global alignment, ensuring well-reconstructed in majority of areas. Secondly, the recursive refinement network further addresses locally misaligned areas, reconstructing HDR frames from bottom to top and recursively refining them to yield faithful reconstruction results. Extensive experimental results demonstrate that our method generates the HDR video with fine details and superior visually, surpassing the state-of-the-art method across diverse scenes.

Global and Local Dual Fusion Network for Large-Ratio Cloud Occlusion Missing Information Reconstruction of a High-Resolution Remote Sensing Image

Global and Local Dual Fusion Network for Large-Ratio Cloud Occlusion Missing Information Reconstruction of a High-Resolution Remote Sensing Image

A Bayesian generative neural network framework for epidemic inference problems

The reconstruction of missing information in epidemic spreading on contact networks can be essential in the prevention and containment strategies. The identification and warning of infectious but asymptomatic individuals (i.e., contact tracing), the well-known patient-zero problem, or the inference of the infectivity values in structured populations are examples of significant epidemic inference problems. As the number of possible epidemic cascades grows exponentially with the number of individuals involved and only an almost negligible subset of them is compatible with the observations (e.g., medical tests), epidemic inference in contact networks poses incredible computational challenges. We present a new generative neural networks framework that learns to generate the most probable infection cascades compatible with observations. The proposed method achieves better (in some cases, significantly better) or comparable results with existing methods in all problems considered both in synthetic and real contact networks. Given its generality, clear Bayesian and variational nature, the presented framework paves the way to solve fundamental inference epidemic problems with high precision in small and medium-sized real case scenarios such as the spread of infections in workplaces and hospitals.

Spatiotemporal Reconstruction of MODIS Normalized Difference Snow Index Products Using U-Net with Partial Convolutions

Moderate Resolution Imaging Spectroradiometer (MODIS) snow cover product is one of the prevailing datasets for global snow monitoring, but cloud obscuration leads to the discontinuity of ground coverage information in spatial and temporal. To solve this problem, a novel spatial-temporal missing information reconstruction model based on U-Net with partial convolutions (PU-Net) is proposed to recover the cloud gaps in the MODIS Normalized Difference Snow Index (NDSI) products. Taking the Yellow River Source Region as a study case, in which the snow cover is characterized by shallow, fast-changing and complex heterogeneity, the MODIS NDSI product in the 2018–2019 snow season is reconstructed, and the reconstruction accuracy is validated with simulated cloud mask and in situ snow depth (SD) observations. The results show that under the simulated cloud mask scenario, the mean absolute error (MAE) of the reconstructed missing pixels is from 4.22% to 18.81% under different scenarios of the mean NDSI of the patch and the mask ratio of the applied mask, and the coefficient of determination (R2) ranges from 0.76 to 0.94. The validation based on in situ SD observations at 10 sites shows good consistency, the overall accuracy is increased by 25.66% to 49.25% compared with the Aqua-Terra combined MODIS NDSI product, and its value exceeds 90% at 60% of observation stations.

Multiple imputation using the average code from autoencoders

Background: Missing information is a constant issue in the clinical setting. The presence of missing values (MV) is triggered by the wrong acquisition of data or sudden events in the patient’s health condition. Imputation arises to replace the non-existent information with the twofold purpose of benefiting from existing information and reducing bias in clinical settings. Mechanisms based on deep learning and multiple imputation (MI) are leading alternatives to impute MVs because of their capacity to extract complex relationships and the consideration of uncertainty that MI adds. Objective: This study aims to improve the reconstruction of missing information through a novel imputation alternative that integrates a MI paradigm into deep learning models. Methods: The proposed method integrates the MI paradigm into the latent representations of an autoencoder, the so-called codes. The average code is then computed, boosting a better latent representation of data. Finally, the average code is decoded to reconstruct MVs. Results: The proposed method is tested in 6 datasets with different patters of MVs. It is compared with solutions based on autoencoders and generative adversarial networks. For the random appearance of MVs, the proposed method outperforms 97% of the scenarios with a reconstruction gain that ranges 1.04-1.45. For the other MVs mechanisms, the proposed method improves the reconstruction in at least 69% of the experiments, with a gain of 1.13-1.91. Conclusion: The findings of the proposed approach showed that the reconstructive capacity of the average code outperforms in most of the scenarios its competitors and close to the best solution in the rest of the scenarios. The integration of the MI paradigm into latent representations of data and the computation of average codes allow a more robust representation of the data and enables the enhancement of current state-of-the-art methods for high MVs rates.

Unsupervised missing information reconstruction for single remote sensing image with Deep Code Regression

Remote sensing images have been applied to many aspects in Earth observation work. However, tons of optical remote sensing images are abandoned due to the information loss caused by the clouds and damage of sensing instruments. Recently, many deep learning methods have been proposed to reconstruct the missing information of remote sensing images but they will be non-effective when it comes to the condition where there is no training dataset. In this paper, we propose an unsupervised method which can reconstruct single remote sensing image without training datasets in a deep neural network. The main idea is to process a reference image of the corrupted image with a deep self-regression network and extract the internal map, which possesses the same spatial information as the reference image. The residual information of the corrupted image is used to constrain the spectral authority of internal map to obtain the reconstruction results. We apply the proposed method in three conditions: 1) dead pixel reconstruction, 2) multitemporal reconstruction and 3) heterogeneous data reconstruction. We conduct simulation experiments and real data experiments in three conditions to confirm the superiority of our methods. The results show that the proposed method outperforms some state-of-the-art algorithms.

Evaluation of Five Deep Learning Models for Crop Type Mapping Using Sentinel-2 Time Series Images with Missing Information

Accurate crop type maps play an important role in food security due to their widespread applicability. Optical time series data (TSD) have proven to be significant for crop type mapping. However, filling in missing information due to clouds in optical imagery is always needed, which will increase the workload and the risk of error transmission, especially for imagery with high spatial resolution. The development of optical imagery with high temporal and spatial resolution and the emergence of deep learning algorithms provide solutions to this problem. Although the one-dimensional convolutional neural network (1D CNN), long short-term memory (LSTM), and gate recurrent unit (GRU) models have been used to classify crop types in previous studies, their ability to identify crop types using optical TSD with missing information needs to be further explored due to their different mechanisms for handling invalid values in TSD. In this research, we designed two groups of experiments to explore the performances and characteristics of the 1D CNN, LSTM, GRU, LSTM-CNN, and GRU-CNN models for crop type mapping using unfilled Sentinel-2 (Sentinel-2) TSD and to discover the differences between unfilled and filled Sentinel-2 TSD based on the same algorithm. A case study was conducted in Hengshui City, China, of which 70.3% is farmland. The results showed that the 1D CNN, LSTM-CNN, and GRU-CNN models achieved acceptable classification accuracies (above 85%) using unfilled TSD, even though the total missing rate of the sample values was 43.5%; these accuracies were higher and more stable than those obtained using filled TSD. Furthermore, the models recalled more samples on crop types with small parcels when using unfilled TSD. Although LSTM and GRU models did not attain accuracies as high as the other three models using unfilled TSD, their results were almost close to those with filled TSD. This research showed that crop types could be identified by deep learning features in Sentinel-2 dense time series images with missing information due to clouds or cloud shadows randomly, which avoided spending a lot of time on missing information reconstruction.

Recurrent-based regression of Sentinel time series for continuous vegetation monitoring

Dense time series of optical satellite imagery describing vegetation activity provide essential information for the efficient and regular monitoring of vegetation. Nevertheless, the temporal resolution of optical sensors is strongly affected by cloud cover, resulting in significant missing information. The use of complementary acquisitions, such as Synthetic Aperture Radar (SAR) data, opens the door to the development of new multi-sensor methodologies aiming at the reconstruction of missing information. However, the joint exploitation of new radar and optical missions, such as the Sentinel, raises new challenges given the different nature and response of the two data sources. In this work, the SenRVM methodology is proposed as a new multi-sensor approach to regress SAR time series towards Normalized Difference Vegetation Index (NDVI). A deep Recurrent Neural Network architecture which integrates SAR acquisitions and ancillary data is adopted. The regression task permits a continuous optical temporal resolution of 6 days. Multiple experiments are carried out to assess the SenRVM framework by studying two large-scale areas in France. Through an extensive interpretation of the results, SenRVM is evaluated on three main vegetation types (grasslands, crops, and forests). High accurate results (R2 > 0.83 and MAE < 0.05) over more than 140,000 vegetation polygons are obtained, both for multi-class and class-specific models. The importance of the SenRVM input features is discussed through an ablation study, highlighting that the relevance of the features differs for the various classes. The good performances of SenRVM are assessed by evaluating the quality of the reconstruction over short- and long-term data gaps. Results are compared to five state-of-the-art methodologies. Finally, some preliminary experiments are first carried out to show how SenRVM results could be used to improve the existing cloud & shadow masks. The recovering of time series breaks caused by vegetation cover changes is also illustrated.

Simultaneous Cloud Detection and Removal From Bitemporal Remote Sensing Images Using Cascade Convolutional Neural Networks

Clouds and cloud shadows heavily affect the quality of the remote sensing images and their application potential. Algorithms have been developed for detecting, removing, and reconstructing the shaded regions with the information from the neighboring pixels or multisource data. In this article, we propose an integrated cloud detection and removal framework using cascade convolutional neural networks, which provides accurate cloud and shadow masks and repaired images. First, a novel fully convolutional network (FCN), embedded with multiscale aggregation and the channel-attention mechanism, is developed for detecting clouds and shadows from a cloudy image. Second, another FCN, with the masks of the detected cloud and shadow, the cloudy image, and a temporal image as the input, is used for the cloud removal and missing-information reconstruction. The reconstruction is realized through a self-training strategy that is designed to learn the mapping between the clean-pixel pairs of the bitemporal images, which bypasses the high demand of manual labels. Experiments showed that our proposed framework can simultaneously detect and remove the clouds and shadows from the images and the detection accuracy surpassed several recent cloud-detection methods; the effects of image restoring outperform the mainstream methods in every indicator by a large margin. The data set used for cloud detection and removal is made open.

Reconstruction Of Missing Information In Satellite Imagery Using STS-CNN

Reconstruction of missing information in any satellite imagery is very important. Without correct information it is very difficult to identify the area in remote sensing images. In general, there are three different types of noises in satellite imagery such as dead pixels and thick cloud cover. The Failure in Instruments such as Aqua MODIS BAND-6 instrument and ETM+ SLC (Scan Line Corrector)-off condition will lead to missing of information. The existing model consists of Double weighted low rank model Algorithm which is a time taking and less efficient model. In this paper, reconstruction of information in satellite image got done using STS-CNN (Spatial-Temporal-Spectral Convolution Neural Network), which is very efficient and uses faster network for reconstruction of information in satellite imagery.

Remote Sensing Image Reconstruction Using Tensor Ring Completion and Total Variation

Time-series remote sensing (RS) images are often corrupted by various types of missing information such as dead pixels, clouds, and cloud shadows that significantly influence the subsequent applications. In this paper, we introduce a new low-rank tensor decomposition model, termed tensor ring (TR) decomposition, to the analysis of RS data sets and propose a TR completion method for the missing information reconstruction. The proposed TR completion model has the ability to utilize the low-rank property of time-series RS images from different dimensions. To further explore the smoothness of the RS image spatial information, total-variation regularization is also incorporated into the TR completion model. The proposed model is efficiently solved using two algorithms, the augmented Lagrange multiplier (ALM) and the alternating least square (ALS) methods. The simulated and real-data experiments show superior performance compared to other state-of-the-art low-rank related algorithms.

Reconstruction of missing information in diffraction patterns and holograms by iterative phase retrieval

It is demonstrated that an object distribution can be successfully retrieved from its diffraction pattern or hologram, even if some of the measured intensity samples are missing. The maximum allowable number of missing values depends on the linear oversampling ratio σ, where the higher the value of σ, the more intensity samples can be missing. For a real-valued object, the ratio of missing pixels to the total number of pixels should not exceed (1 - 2/σ2) or (1 - 1/σ2) in the acquired diffraction pattern or hologram, respectively. For example, even 5% of the measured intensity values at an oversampling ratio of σ=8 are sufficient to simultaneously retrieve the object distribution and the missing intensity values. It is important that the missing intensity values should not be concentrated in the centre, but should be randomly distributed over the acquired diffraction pattern.

Pansharpening for Cloud-Contaminated Very High-Resolution Remote Sensing Images

The optical remote sensing images not only have to make a fundamental tradeoff between the spatial and spectral resolutions, but also are inevitable to be polluted by the clouds; however, the existing pansharpening methods mainly focus on the resolution enhancement of the optical remote sensing images without cloud contamination. How to fuse the cloud-contaminated images to achieve the joint resolution enhancement and cloud removal is a promising and challenging work. In this paper, a pansharpening method for the challenging cloud-contaminated very high-resolution remote sensing images is proposed. Furthermore, the cloud-contaminated conditions for the practical observations with all the thick clouds, the thin clouds, the haze, and the cloud shadows are comprehensively considered. In the proposed methods, a two-step fusion framework based on multisource and multitemporal observations is presented: 1) the thin clouds, the haze, and the light cloud shadows are proposed to be first jointly removed and 2) a variational-based integrated fusion model is then proposed to achieve the joint resolution enhancement and missing information reconstruction for the thick clouds and dark cloud shadows. Through the proposed fusion method, a promising cloud-free fused image with both high spatial and high spectral resolutions can be obtained. To comprehensively test and verify the proposed method, the experiments were implemented based on both the cloud-free and cloud-contaminated images, and a number of different remote sensing satellites including the IKONOS, the QuickBird, the Jilin (JL)-1, and the Deimos-2 images were utilized. The experimental results confirm the effectiveness of the proposed method.

Missing Data Reconstruction for Remote Sensing Images With Weighted Low-Rank Tensor Model

Missing data reconstruction for remote sensing images, such as dead-pixel recovery and cloud removal, is important for remote sensing data applications. Missing information reconstruction is well known as being an ill-posed inverse problem. In this paper, a weighted low-rank tensor regularization model is proposed to handle the problem. The proposed model fully utilizes the correlations in the spatial, spectral, and temporal components of the remote sensing images to adaptively deal with the varied missing data problems, including the Aqua Moderate Resolution Imaging Spectroradiometer (MODIS) band 6 dead line problem, the Landsat scan-line corrector failure (SLC-off) problem, and cloud contamination. A double-weighted treatment is developed to balance the contributions from the different dimensions and preserve the different structures and textures in remote sensing images. The experiments undertaken confirmed the good performance of the proposed method, and the reconstruction results of the proposed method, in both visual effect and quantitative evaluation, were superior to those of the other methods.

Robust Image Completion via Deep Feature Transformations

For many practical applications, it is essential to address both geometric corrections and missing information reconstruction of face images and natural images. However, it is unfavorable to separate the problem into two sub-tasks due to error accumulations of sequential tasks. In this paper, we propose a novel robust missing information reconstruction framework via deep feature transformations to simultaneously address both geometric corrections and image completion. Specifically, our proposed framework realizes multiple channel spatial transformations to tackle geometric corrections, and address image completion through non-linear features projections. The flow of our framework includes deep feature extraction, feature enhancement, feature projection, and feature refinement, where deep features are extracted and learnt to achieve robust image completion. Experimental results show the superior performance of our framework for both face images and natural images in various databases. Compared with the conventional approaches approach to split the problem into two sub-tasks, including image inpainting and spatial transformation, our proposed framework achieves a number of advantages, including i) an unified framework to automatically correct the geometric distortions and to reconstruct the missing information simultaneously and ii) achieving much better visual quality for those recovered images.

Missing Data Reconstruction in Remote Sensing Image With a Unified Spatial–Temporal–Spectral Deep Convolutional Neural Network

Because of the internal malfunction of satellite sensors and poor atmospheric conditions such as thick cloud, the acquired remote sensing data often suffer from missing information, i.e., the data usability is greatly reduced. In this paper, a novel method of missing information reconstruction in remote sensing images is proposed. The unified spatial-temporal-spectral framework based on a deep convolutional neural network (STS-CNN) employs a unified deep convolutional neural network combined with spatial-temporal-spectral supplementary information. In addition, to address the fact that most methods can only deal with a single missing information reconstruction task, the proposed approach can solve three typical missing information reconstruction tasks: 1) dead lines in Aqua MODIS band 6; 2) the Landsat ETM+ Scan Line Corrector (SLC)-off problem; and 3) thick cloud removal. It should be noted that the proposed model can use multi-source data (spatial, spectral, and temporal) as the input of the unified framework. The results of both simulated and real-data experiments demonstrate that the proposed model exhibits high effectiveness in the three missing information reconstruction tasks listed above.

Cloud removal for hyperspectral remotely sensed images based on hyperspectral information fusion

ABSTRACTHyperspectral remote sensing plays an important role in a wide variety of fields. However, its specific application for land surface analysis has been constrained due to the different shapes of thick, opaque cloud cover. The reconstruction of missing information obscured by clouds in remote-sensing images is an area of active research. However, most of the available cloud-removal methods are not suitable for hyperspectral images, because they lose the spectral information which is very important for hyperspectral analysis. In this article, we developed a new spectral resolution enhancement method for cloud removal (SREM-CR) from hyperspectral images, with the help of an auxiliary cloud-free multispectral image acquired at different times. In the fixed hyperspectral image, spectra of the cloud cover pixels are reconstructed depending on the relationship between the original hyperspectral and multispectral images. The final resulting image has the same spectral resolution as the original hyperspectral image but without clouds. This approach was tested on two experiments, in which the results were compared by visual interpretation and statistical indices. Our method demonstrated good performance.

Multiscale adaptive reconstruction of missing information for remotely sensed data using sparse representation

ABSTRACTDue to the influence of sensor malfunction and poor atmospheric condition, missing information is inevitable in optical remotely sensed (RS) data, which limits the availability of RS data. To tackle the inverse problem of missing information recovery, a multiscale adaptive patch reconstruction method was proposed in this letter. Multiscale dictionaries were learned from different sizes of exemplars in the known image region, which were later utilized to infer missing information patch-by-patch via sparse representation. Structure sparsity was incorporated to encourage the filling-in of missing patch on image structures and determine the patch size for further inpainting. Neighboring information was employed to restrain the appearance of the estimated patch, to yield semantically consistent inpainting result. In view of these ideas, we formulate the optimization model of adaptive patch inpainting and reconstruct missing information through a multiscale scheme. Experiments are performed on cloud removal, gaps filling and quantitative product reconstruction, which demonstrate that our method can well preserve spatially continuous structures and consistent textures without artifacts.

Color Channel Reconstruction for Multi-Color Multi-View Images Using Disparity and Color Similarity-based Local Linear Regression

Most digital cameras today employ Bayer Color Filter Arrays in front of the camera sensor. In order to create a true-color image, a demosaicing step is required introducing image blur and artifacts. Special sensors like the Foveon X3 circumvent the demosaicing challenge by using pixels lying on top of each other. However, they are not commonly used due to high production cost and low flexibility. In this work, a multi-color multi-view approach is presented in order to create true-color images. Therefore, the red-filtered left view and the blue-filtered right view are registered and projected onto the green-filtered center view. Due to the camera offset and slightly different viewing angles of the scene, object occlusions might occur for the side channels, hence requiring the reconstruction of missing information. For that, a novel local linear regression method is proposed, based on disparity and color similarity. Simulation results show that the proposed method outperforms existing reconstruction techniques by on average 5 dB.

Missing Information Reconstruction of Remote Sensing Data: A Technical Review

Because of sensor malfunction and poor atmospheric conditions, there is usually a great deal of missing information in optical remote sensing data, which reduces the usage rate and hinders the follow-up interpretation. In the past decades, missing information reconstruction of remote sensing data has become an active research field, and a large number of algorithms have been developed. However, to the best of our knowledge, there has not, to date, been a study that has been aimed at expatiating and summarizing the current situation. This is therefore our motivation in this review. This paper provides an introduction to the principles and theories of missing information reconstruction of remote sensing data. We classify the established and emerging algorithms into four main categories, followed by a comprehensive comparison of them from both experimental and theoretical perspectives. This paper also predicts the promising future research directions.