Reconstruction Of Missing Data Research Articles

Deep learning tool: reconstruction of long missing climate data based on spatio-temporal multilayer perceptron

AbstractLong-term monitoring of climate data is significant for grasping the law and development trend of climate change and guaranteeing food security. However, some weather stations lack monitoring data for even decades. In this study, 62 years of historical monitoring data from 105 weather stations in Xinjiang were used for missing sequence prediction, validating proposed data reconstruction tool. First of all, study area was divided into three parts according to the climatic characteristics and geographical locations. A spatio-temporal multilayer perceptron (MLP) was established to reconstruct meteorological data with three time scales (Short term, cycle and long term) and one spatio dimension as inputing (rolling predictions, one step predicts one day), filling in long sequence blank data. By designing an end-to-end model to autonomously detect the locations of missing data and make rolling predictions,we obtained complete meteorological monitoring data of Xinjiang from 1961 to 2022. Seven kinds of parameter reconstructed include maximum temperature (Max_T), minimum temperature (Min_T), mean temperature (Ave _ T), average water vapor pressure (Ave _ WVP), relative humidity (Ave _ RH), average wind speed (10 m Ave _ WS), and sunshine duration (Sun_H). Contrasted the prediction accuracy of the model with general MLP and LSTM, results shows that, in the seven types of parameters, designed spatio-temporal MLP decreases MAE and MSE by 7.61% and 4.80% respectively. The quality of reconstructed data was evaluated by calculating correlation coefficient with the monitored sequences of nearest station,determining the applicable meteorological parameters of the model according to the results. Results show that,proposed model reached satisfied average correlation coefficient for Max_T, Min_T, Ave _ T and Ave _ WVP parameters are 0.969, 0.961, 0.971 and 0.942 respectively. The average correlation coefficient of Sun_H and Ave _ RH are 0.720 and 0.789. Although it is difficult to predict extreme values, it can still capture the period and trend; the reconstruction effect of 10 m Ave _ WS is poor, with the average similarity of 0.488. Proposed method is applicable to reconstruct Max_T, Min_T, Ave _ T and Ave _ WVP, but not recommended to reconstruct Sun_H, Ave _ RH and Ave _ WS.

A Novel Method for Missing Data Reconstruction in Smart Grid Using Generative Adversarial Networks

A Novel Method for Missing Data Reconstruction in Smart Grid Using Generative Adversarial Networks

The Effectiveness of two Dimensional Haar Wavelet Image De-noising performance using Soft or Hard Thresholding Approach

Image de-noising and restoration represent basic problems in image processing with many different applications including engineering, reconstruction of missing data during their transmission and enhancement ..etc. this work is aimed at developing effective algorithm for denoising image using new strategy algorithm of wavelet techniques ,by applying two dimensions wavelet transform using Haar wavelet. Wavelets are hierarchically decomposing mathematical tools. A noisy picture is sent to the Haar wavelet transform to create four decomposed bands(for each level), and the noise is then removed by applying a threshold to the resultant image band. Then, an inverse transform is used to provide a noise-reduced picture. Performance characteristics are improved by the suggested design in contrast to those of the current approaches.

Solar and Wind Data Recognition: Fourier Regression for Robust Recovery

Accurate prediction of renewable energy output is essential for integrating sustainable energy sources into the grid, facilitating a transition towards a more resilient energy infrastructure. Novel applications of machine learning and artificial intelligence are being leveraged to enhance forecasting methodologies, enabling more accurate predictions and optimized decision-making capabilities. Integrating these novel paradigms improves forecasting accuracy, fostering a more efficient and reliable energy grid. These advancements allow better demand management, optimize resource allocation, and improve robustness to potential disruptions. The data collected from solar intensity and wind speed is often recorded through sensor-equipped instruments, which may encounter intermittent or permanent faults. Hence, this paper proposes a novel Fourier network regression model to process solar irradiance and wind speed data. The proposed approach enables accurate prediction of the underlying smooth components, facilitating effective reconstruction of missing data and enhancing the overall forecasting performance. The present study focuses on Midland, Texas, as a case study to assess direct normal irradiance (DNI), diffuse horizontal irradiance (DHI), and wind speed. Remarkably, the model exhibits a correlation of 1 with a minimal RMSE (root mean square error) of 0.0007555. This study leverages Fourier analysis for renewable energy applications, with the aim of establishing a methodology that can be applied to a novel geographic context.

Enhancing Rainfall Data Consistency and Completeness: A Spatiotemporal Quality Control Approach and Missing Data Reconstruction Using MICE on Large Precipitation Datasets

Enhancing Rainfall Data Consistency and Completeness: A Spatiotemporal Quality Control Approach and Missing Data Reconstruction Using MICE on Large Precipitation Datasets

Using complex network theory for missing data reconstruction in water distribution networks

Dealing with missing data is a critical challenge in analyzing water distribution networks (WDNs). These unknown data are often associated with the physical characteristics of pipes, like diameter, which are the foundations of several modeling endeavors. This study develops a fully automated model to reconstruct missing diameter information in WDNs within the complex network theory (CNT) framework. The newly developed model employs customized graph measures to systematically retrieve missing diameters based on the topological (e.g., connectivity) and hydraulic (e.g., flow) features of edges with available information. The proposed model is validated by testing it on three real-world WDNs located in Iran and Austria, where data gaps are created by randomly and progressively eliminating the pipe diameter information. Subsequently, the model's accuracy is assessed by comparing the diameter, pressure, and hydraulic-based resilience of the reconstructed WDNs with those of the complete dataset. The results indicate that the proposed reconstruction model can successfully retrieve missing information up to 90% of the data gap. In addition, the model is highly computationally efficient and can quickly reconstruct thousands of missing diameters. Our novel approach can benefit water utilities that frequently encounter incomplete data in their models and require missing data retrieval from large-scale WDNs.

An enhanced feature extraction based long short-term memory neural network for wind power forecasting via considering the missing data reconstruction

Wind power forecasting plays a significant role in regulating the peak and frequency of the power system, which can improve the wind power receiving capacity. Despite plenty of forecasting methods have been proposed to fortify the accuracy of forecasting, the existing forecasting models do not consider the reconstruction of missing data and can not extract the spatiotemporal features from the wind power data. To address these issues, this study proposes an improved long short-term memory (LSTM) network based forecasting method to reconstruct the missing data and capture the spatiotemporal features from wind power data. In order to fortify the wind power forecasting accuracy of the model, the multiple imputation technique (MIT) is first developed to fill up the missing samples with reconstructed data samples by analyzing the correlation among variables in the raw wind power data. Secondly, to exploit the spatial and temporal features and reduce the low computation complexity, a new parallel convolutional network involving dilated convolution and causal convolution is established for features extraction. Finally, to further improve the wind power forecasting performance, the LSTM is applied to extract the long-term trends and reveal the internal relations of the derived spatiotemporal features. The experimental results on a benchmark dataset and an experimental dataset both demonstrate that the proposed LSTM based wind power forecasting can obtain the better forecasting performance.

Performance of matrix completion approaches for aquaponics data

Technological innovations in Internet of Things (IoT) have resulted in smart agricultural solutions such as a remotely monitored Aquaponics system and a wireless sensor network (WSN) of such systems (nodes). IoT enables continuous sensing of temperature and pH data at each node of the WSN, which is periodically transmitted to a remote fusion centre. In this regard, the data matrices acquired at the fusion centre often suffer from data vacancies and missing data problems, owing to typical wireless multipath fading environment, sensor malfunctions and node failures. This paper explores the applicability of different matrix completion approaches for missing data reconstruction. Specifically, the performance of baseline predictor, correlation based approaches such as baseline predictor with temporal model, k-nearest neighbors (kNN) and low rank based approaches such as Sparsity Regularized Singular Value Decomposition (SRSVD) and Augmented Lagrangian Sparsity Regularized Matrix Factorization (ALSRMF) have been explored. Reliable temperature and pH data for 19 independent acquisition hours with 60 samples per hour are acquired at the fusion centre via Ultra High Frequency (UHF) transmission at 470 MHz and suitable pre-processing. Simulating different data integrity scenarios, the reconstruction error plots from each of these matrix completion approaches is extracted. A hybrid of kNN and baseline predictor with temporal model rendered a Mean Absolute Percentage Error (MAPE) of 1.75% for temperature and 0.86% for pH, at 0.5 data integrity. Further, with ALSRMF, which exploits the low rank constraint, the error reduced to 1.25% for temperature and 0.7% for pH, thus substantiating a promising approach for Aquaponics system data reconstruction.

Multiple-layer statistical methodology for developing data-driven models of anaerobic digestion process

When modelling anaerobic digestion, ineffective data handling and inadequate designation of modelling parameters can undermine the model reliability. In this study, a multilayer statistical technique, which employed a machine learning technique using regression models, was introduced to systematically support the development of anaerobic digestion models. Layer-by-layer statistical techniques including cubic smoothing splines (missing data reconstruction), principal component analysis (identifying correlated parameters), analysis of variance (analysing differences among datasets), and linear regression (developing data-driven models) were used to develop and validate anaerobic digestion models. Experimental data collected from the long-term operation of lab-scale (operated for 350 days), pilot-scale (operated for 150 days), and full-scale reactors (operated for 750 days) were used to demonstrate the modelling process. The multivariate models based on a data-driven modelling technique were developed by subjecting the experimental and monitored data to a modelling process. The developed models could predict the biogas production and effluent chemical oxygen demand during anaerobic digestion. Statistical analyses verified the modelling hypotheses, evaded invalid model development, and ensured data integrity and parameter validity. Multiple linear regression of principal components demonstrated that the performance of biogas production using food waste was influenced by the variances of the nitrogen and organic concentrations, but not by the chemical oxygen demand to total nitrogen (C/N) ratio. In the validation process, the model developed with lab-scale reactor data showed relatively high accuracy with R2, SSE, and RMSE values of 0.86, 34.45, and 0.72.

Reconstruction of daily chlorophyll-a concentrations in the transit of severe tropical cyclone Hudhud using the ExDINEOF method

Tropical regions experience a diverse range of dense clouds, posing challenges for the daily reconstruction of chlorophyll-a concentration data. This underscores the pressing need for a practical method to reconstruct daily-scale chlorophyll-a concentrations in such regions. While traditional data reconstruction methods focus on single variables and rely on specific factors to infer missing data at specific locations, these single-variable methods may falter when applied to tropical oceans due to the scarcity of available data. Fortunately, all oceanographic variables undergo similar atmospheric and marine dynamic processes, creating internal relationships between them. This allows for the reconstruction of missing data through correlations between variables. Thus, this study introduces a multivariate reconstruction approach using the extended data interpolating empirical orthogonal function (ExDINEOF) method to reconstruct missing daily-scale chlorophyll-a concentration data. The ExDINEOF method considers the simultaneous relationships among multiple variables for data reconstruction in tropical oceans. To verify the method’s robustness, missing data were reconstructed during the formation and passage of severe tropical cyclone Hudhud through the Bay of Bengal. The results demonstrate that ExDINEOF outperforms traditional data reconstruction methods, exhibiting favorable spatial distribution and enhanced accuracy within the dynamic tropical marine environment. Furthermore, an assessment of marine physical environmental factors associated with chlorophyll-a concentration data provides additional evidence for the ExDINEOF method’s accuracy. Notably, the ExDINEOF method offers comprehensive spatial distribution aligned with underlying physical mechanisms governing phytoplankton distribution patterns, detailed phytoplankton growth, bloom, extinction variations in time series, satisfactory accuracy, and comprehensive local-level details.

Optimal sensor placement for reconstructing wind pressure field around buildings using compressed sensing

Deciding how to optimally deploy sensors in a large, complex, and spatially extended structure is critical to ensure that the surface pressure field is accurately captured for subsequent analysis and design. In some cases, reconstruction of missing data is required in downstream tasks such as the development of digital twins. This paper presents a data-driven sparse sensor selection algorithm, aiming to provide the most information contents for reconstructing aerodynamic characteristics of wind pressures over tall building structures parsimoniously. The algorithm first fits a set of basis functions to the training data, then applies a computationally efficient QR algorithm that ranks existing pressure sensors in order of importance based on the state reconstruction to this tailored basis. The findings of this study show that the proposed algorithm successfully reconstructs the aerodynamic characteristics of tall buildings from sparse measurement locations, generating stable and optimal solutions across a range of conditions. As a result, this study serves as a promising first step toward leveraging the success of data-driven and machine learning algorithms to supplement traditional genetic algorithms currently used in wind engineering.

Uncertainty Propagation of Missing Data Signals with the Interval Discrete Fourier Transform

The interval discrete Fourier transform (DFT) algorithm can propagate signals carrying interval uncertainty. By addressing the repeated variables problem, the interval DFT algorithm provides exact theoretical bounds on the Fourier amplitude and estimates of the power spectral density (PSD) function while running in polynomial time. Thus, the algorithm can be used to assess the worst-case scenario in terms of maximum or minimum power, and provide insights into the amplitude spectrum bands of the transformed signal. To propagate signals with missing data, an upper and lower value for the missing data present in the signal must be assumed, such that the uncertainty in the spectrum bands can also be interpreted as an indicator of the quality of the reconstructed signal. For missing data reconstruction, there are a number of techniques available that can be used to obtain reliable bounds in the time domain, such as Kriging regressors and interval predictor models. Alternative heuristic strategies based on variable—as opposed to fixed—bounds can also be explored. This work aims to investigate the sensitivity of the algorithm against interval uncertainty in the time signal. The investigation is conducted in different case studies using signals of different lengths generated from the Kanai-Tajimi PSD function, representing earthquakes, and the Joint North Sea Wave Observation Project (JONSWAP) PSD function, representing sea waves as a narrowband PSD model.

Reconstruction of missing data of the PandaX-IIIexperiment TPC using neural network

The PandaX-III is a Neutrinoless Double-Beta Decay (NLDBD) experiment which uses a Time Projection Chamber (TPC) detector with a readout plane formed of Micromegas modules, which allows reconstruction of track topology for the background discrimination as well as reconstruction of the energy of the events. In NLDBD experiments, in order to achieve the highest sensitivity to the decay, it is necessary for the detector to have a high energy resolution, the background level should be low, and techniques for background discrimination must be applied as well. In reality, inhomogeneous signal gain at each module and the presence of missing channels lead to an incorrect energy reconstruction of the events. In this work, a method based on a Convolutional Neural Networks (CNN) aiming to reconstruct the kinematics of the event from imperfect data with missing channels is presented. Preliminary results of the reconstruction of the missing data using CNN are showing an increase in detection efficiency. The detection efficiency was evaluated on the simulated data with three channels randomly chosen per Micromegas module and artificially set as missing. Direct reconstruction of the energy gives the efficiency of 78%, while after applying CNN it increases to 86%, providing a promising application of this technique.

Simultaneous off-the-grid regularization and reconstruction for 3D seismic data by a new combined sampling operator

Seismic samples are generally designed to be placed on perfect Cartesian coordinates, that is, on-the-grid. However, sampling geometry is disturbed by obstacles in field applications. Large obstacles result in missing samples. For small obstacles, geophones or sources are placed at an available off-the-grid location nearest to the designed grid. To achieve simultaneous off-the-grid regularization and missing data reconstruction for 3D seismic data, we develop a new mathematical model based on a new combined sampling operator, a 3D curvelet transform, and a fast projection onto convex sets (FPOCS) algorithm. The sampling operator is combined with a binary mask for on-the-grid samples reconstruction and a barycentric Lagrangian (BL) operator for off-the-grid samples regularization. A 2D BL operator is obtained using the tensor product of two 1D BL operators. The inversion problem is efficiently solved based on FPOCS. This method is tested on synthetic and field data sets. The reconstruction results outperform the methods based on the binary mask in terms of signal-to-noise ratio and visual effect.

Pyseistr: A Python Package for Structural Denoising and Interpolation of Multichannel Seismic Data

AbstractNew sensing techniques like the nodal geophones and distributed acoustic sensing enable a spatial sampling ratio that was unprecedentedly high in earthquake seismology. The much higher sampling of seismic wavefields that is close to the level in exploration seismology calls for a unified processing approach for multichannel seismic data regardless of the research interest, for example, oil and gas oriented or earthquake-study oriented. Here, we present the first Python package for multichannel seismic data that benefits both communities, that is, exploration and earthquake seismology, called Pyseistr. The Pyseistr is a Python package that is designed to make full use of the structural patterns in multichannel seismic data to facilitate the data processing. The Pyseistr package currently includes several fundamental functions like slope estimation, structural mean and median filtering, and structural reconstruction of missing data. The Pyseistr package is continuously developed to include more functions that benefit both exploration and earthquake communities.

Application of Machine Learning Techniques to Ocean Mooring Time Series Data

Abstract In situ observations are vital to improving our understanding of the variability and dynamics of the ocean. A critical component of the ocean circulation is the strong, narrow, and highly variable western boundary currents. Ocean moorings that extend from the seafloor to the surface remain the most effective and efficient method to fully observe these currents. For various reasons, mooring instruments may not provide continuous records. Here we assess the application of the Iterative Completion Self-Organizing Maps (ITCOMPSOM) machine learning technique to fill observational data gaps in a 7.5 yr time series of the East Australian Current. The method was validated by withholding parts of fully known profiles, and reconstructing them. For 20% random withholding of known velocity data, validation statistics of the u- and υ-velocity components are R2 coefficients of 0.70 and 0.88 and root-mean-square errors of 0.038 and 0.064 m s−1, respectively. Withholding 100 days of known velocity profiles over a depth range between 60 and 700 m has mean profile residual differences between true and predicted u and υ velocity of 0.009 and 0.02 m s−1, respectively. The ITCOMPSOM also reproduces the known velocity variability. For 20% withholding of salinity and temperature data, root-mean-square errors of 0.04 and 0.38°C, respectively, are obtained. The ITCOMPSOM validation statistics are significantly better than those obtained when standard data filling methods are used. We suggest that machine learning techniques can be an appropriate method to fill missing data and enable production of observational-derived data products. Significance Statement Moored observational time series of ocean boundary currents monitor the full-depth variability and change of these dynamic currents and are used to understand their influence on large-scale ocean climate, regional shelf–coastal processes, extreme weather, and seasonal climate. In this study we apply a machine learning technique, Iterative Completion Self-Organizing Maps (ITCOMPSOM), to fill data gaps in a boundary current moored observational data record. The ITCOMPSOM provides an improved method to fill data gaps in the mooring record and if applied to other observational data records may improve the reconstruction of missing data. The derived gridded data product should improve the accessibility and potentially increase the use of these data.

An Improved DINEOF Algorithm Based on Optimized Validation Points Selection Method

Ocean remote-sensing satellite data have been widely applied in the areas of oceanography, meteorology, the environment, and many more fields in science and engineering. However, missing data due to cloud cover, equipment failure, etc., limit its application. Therefore, reconstruction of the missing data through an appropriate method is essential. The data-interpolating empirical orthogonal function (DINEOF) algorithm proposed by Beckers and Rixen (2003) is currently the most commonly used method for the reconstruction of missing data in large areas. However, the existing DINEOF algorithm adopts a random method to select the cross−validation points, which may underutilize effective information around the missing value points. In addition, the cross-validation points may be too concentrated in an area, thus being unable to reflect the overall characteristics of the data. This paper optimizes the method to select the cross-validation points so that the information around the missing values can be effectively utilized and to avoid the cross-validation points being too concentrated. On this basis, an improved validation-point DINEOF algorithm (IV−DINEOF) is proposed. An ideal dataset and a reanalysis dataset based on sea surface temperature (SST) are used to test the performance of the improved algorithm. Statistical analysis of the results shows that the data reconstruction performance of the IV−DINEOF algorithm is better than that of the DINEOF algorithm, and the computational efficiency is also improved. The VE−DINEOF algorithm has the highest computing efficiency, but its reconstruction accuracy is lower than that of IV−DINEOF.

Reconstruction of missing data in the global surface water dataset with a self-supervised deep learning model

Reconstruction of missing data in the global surface water dataset with a self-supervised deep learning model

Missing data reconstruction in attitude for quadrotor unmanned aerial vehicle based on deep regression model with different sensor failures

Forming a UAV cluster with multiple UAVs to collaborate in accomplishing tasks is the future development of battlefield. The UAV cluster can collaborate by communicating through inter-aircraft links and can quickly and accurately perform complex tasks such as path planning, collaborative reconnaissance, collaborative sensing and attack. In the process of Unmanned Aerial Vehicle (UAV) cooperative operation, obtaining the real-time attitude information of each UAV is preliminary to implement collaborative cooperation. However, in practice, due to the internal malfunction of sensors and mutual interference among UAVs, the acquired attitudes often suffer from data missing. This paper proposed a novel missing data reconstruction method for UAV’s attitude in the case of sensor failures. The attitude data of UAV is obtained through an advanced UAV simulation platform AirSim. Through fusing the temporal convolutional network (TCN) and Bi-directional long short-term memory (Bi-LSTM), a deep regression framework is established for missing data construction. In addition, we evaluate the reliability of the proposed method by comparing with different baseline models and different combinations of data missing. The performance of our method is quantified using metrics of root-mean-square-error (RMSE), mean absolute error (MAE), determination coefficient (DC), mean absolute percentage error (MAPE), and symmetric mean absolute percentage error (SMAPE) score. The averaged performance values of our proposed method outperform other baseline algorithms. The results of the experiment show that the study could well-address the issue of missing data reconstruction and provide reliable attitude data.

Irregularly Spatial Seismic Missing Data Reconstruction Using Transformer With Periodic Skip Connection

Irregularly Spatial Seismic Missing Data Reconstruction Using Transformer With Periodic Skip Connection