Missing Data In Time Series Research Articles

Missing Data Filling of Model Based on Neural Network

Today’s society has entered the pace of information development era. All kinds of information are digitized, while the loss of information data also directly affects the normal operation of the application system and has become the biggest obstacle to the development of information technology. The existing missing filling methods do not take into account the time-series information of the data set. Based on the neural network method for filling missing data in time series, an end-to-end missing data filling method for time series based on the residual of regression equation is proposed. Under the assumption that the activation function and the noise diffusion coefficient function are satisfied, the exponential stability of the complex-valued stochastic inertial neural network systems with additive time-varying delays is studied. The bipartite graph model of the time series data missing value filling method of the countermeasure network and the filling cyclic neural unit are generated. According to the end-to-end time series data missing value filling method of the self encoder, the corresponding low-dimensional feature vector can be automatically generated for each time series data. It avoids overfitting by using nonlinear methods directly in low-dimensional space. Experiments show that the method proposed in this paper uses Monte Carlo to fill data missing at different proportions. In data gap filling process, the coefficient variance is less than 0.05, which enhances the rationality of filling data. It is of great research value and practical significance to reasonably fill in the missing values of time series data. The research can accelerate the improvement of big data system, improve the level and effectiveness of database management, and is an important means of computing capacity of existing data centers.

Missing Data Imputation Method Combining Random Forest and Generative Adversarial Imputation Network.

(1) Background: In order to solve the problem of missing time-series data due to the influence of the acquisition system or external factors, a missing time-series data interpolation method based on random forest and a generative adversarial interpolation network is proposed. (2) Methods: First, the position of the missing part of the data is calibrated, and the trained random forest algorithm is used for the first data interpolation. The output value of the random forest algorithm is used as the input value of the generative adversarial interpolation network, and the generative adversarial interpolation network is used to calibrate the position. The data are interpolated for the second time, and the advantages of the two algorithms are combined to make the interpolation result closer to the true value. (3) Results: The filling effect of the algorithm is tested on a certain bearing data set, and the root mean square error (RMSE) is used to evaluate the interpolation results. The results show that the RMSE of the interpolation results based on the random forest and generative adversarial interpolation network algorithms in the case of single-segment and multi-segment missing data is only 0.0157, 0.0386, and 0.0527, which is better than the random forest algorithm, generative adversarial interpolation network algorithm, and K-nearest neighbor algorithm. (4) Conclusions: The proposed algorithm performs well in each data set and provides a reference method in the field of data filling.

Missing data imputation and classification of small sample missing time series data based on gradient penalized adversarial multi-task learning

Missing data imputation and classification of small sample missing time series data based on gradient penalized adversarial multi-task learning

Gap-filling missing data in time series using the correlation matrix method of multiple time series in Asadabad Plain, Iran

Gap-filling missing data in time series using the correlation matrix method of multiple time series in Asadabad Plain, Iran

An Evolutionary Artificial Neural Network approach for spatio-temporal wave height time series reconstruction

This paper proposes a novel methodology for recovering missing time series data, a crucial task for subsequent Machine Learning (ML) analyses. The methodology is specifically applied to Significant Wave Height (SWH) time series in the field of marine engineering. The proposed approach involves two phases. Firstly, the SWH time series for each buoy is independently reconstructed using three transfer function models: regression-based, correlation-based, and distance-based. The distance-based transfer function exhibits the best overall performance. Secondly, Evolutionary Artificial Neural Networks (EANNs) are utilised for the final recovery of each time series, using as inputs highly correlated buoys that have been intermediately recovered. The EANNs are evolved considering two metrics, the novel squared error relevance area, which balances the importance of extreme and around-mean values, and the well-known mean squared error. The study considers SWH time series data from 15 buoys in two coastal zones in the United States. The results demonstrate that the distance-based transfer function is generally the best transfer function, and that EANNs outperform a range of state-of-the-art ML techniques in 12 out of the 15 buoys, with a number of connections comparable to linear models. Furthermore, the proposed methodology outperforms the two most popular approaches for time series reconstruction, BRITS and SAITS, for all buoys except one. Therefore, the proposed methodology provides a promising approach, which may be applied to time series from other fields, such as wind or solar energy farms in the field of green energy.

Correction to: Multi‑head attention‑based model for reconstructing continuous missing time series data

Correction to: Multi‑head attention‑based model for reconstructing continuous missing time series data

Multi-head attention-based model for reconstructing continuous missing time series data

Multi-head attention-based model for reconstructing continuous missing time series data

Using deep learning to interpolate the missing data in time-series for credit risks along supply chain

PurposeAs the supply chain is a highly integrated infrastructure in modern business, the risks in supply chain are also becoming highly contagious among the target company. This motivates researchers to continuously add new features to the datasets for the credit risk prediction (CRP). However, adding new features can easily lead to missing of the data.Design/methodology/approachBased on the gaps summarized from the literature in CRP, this study first introduces the approaches to the building of datasets and the framing of the algorithmic models. Then, this study tests the interpolation effects of the algorithmic model in three artificial datasets with different missing rates and compares its predictability before and after the interpolation in a real dataset with the missing data in irregular time-series.FindingsThe algorithmic model of the time-decayed long short-term memory (TD-LSTM) proposed in this study can monitor the missing data in irregular time-series by capturing more and better time-series information, and interpolating the missing data efficiently. Moreover, the algorithmic model of Deep Neural Network can be used in the CRP for the datasets with the missing data in irregular time-series after the interpolation by the TD-LSTM.Originality/valueThis study fully validates the TD-LSTM interpolation effects and demonstrates that the predictability of the dataset after interpolation is improved. Accurate and timely CRP can undoubtedly assist a target company in avoiding losses. Identifying credit risks and taking preventive measures ahead of time, especially in the case of public emergencies, can help the company minimize losses.

Analyzing the Performance of Transformers for the Prediction of the Blood Glucose Level Considering Imputation and Smoothing

In this paper we investigate the effect of two preprocessing techniques, data imputation and smoothing, in the prediction of blood glucose level in type 1 diabetes patients, using a novel deep learning model called Transformer. We train three models: XGBoost, a one-dimensional convolutional neural network (1D-CNN), and the Transformer model to predict future blood glucose levels for a 30-min horizon using a 60-min time series history in the OhioT1DM dataset. We also compare four methods of handling missing time series data during the model training: hourly mean, linear interpolation, cubic interpolation, and spline interpolation; and two smoothing techniques: Kalman smoothing and smoothing splines. Our experiments show that the Transformer performs better than XGBoost and 1D-CNN when only continuous glucose monitoring (CGM) is used as a predictor, and that it is very competitive against XGBoost when CGM and carbohydrate intake from the meal are used to predict blood glucose level. Overall, our results are more accurate than those appearing in the literature.

Mind the Large Gap: Novel Algorithm Using Seasonal Decomposition and Elastic Net Regression to Impute Large Intervals of Missing Data in Air Quality Data

Air quality data sets are widely used in numerous analyses. Missing values are ubiquitous in air quality data sets as the data are collected through sensors. Recovery of missing data is a challenging task in the data preprocessing stage. This task becomes more challenging in time series data as time is an implicit variable that cannot be ignored. Even though existing methods to deal with missing data in time series perform well in situations where the percentage of missing values is relatively low and the gap size is small, their performances are reasonably lower when it comes to large gaps. This paper presents a novel algorithm based on seasonal decomposition and elastic net regression to impute large gaps of time series data when there exist correlated variables. This method outperforms several other existing univariate approaches namely Kalman smoothing on ARIMA models, Kalman smoothing on structural time series models, linear interpolation, and mean imputation in imputing large gaps. However, this is applicable only when there exists one or more correlated variables with the time series with large gaps.

Binned Data Provide Better Imputation of Missing Time Series Data from Wearables.

The presence of missing values in a time-series dataset is a very common and well-known problem. Various statistical and machine learning methods have been developed to overcome this problem, with the aim of filling in the missing values in the data. However, the performances of these methods vary widely, showing a high dependence on the type of data and correlations within the data. In our study, we performed some of the well-known imputation methods, such as expectation maximization, k-nearest neighbor, iterative imputer, random forest, and simple imputer, to impute missing data obtained from smart, wearable health trackers. In this manuscript, we proposed the use of data binning for imputation. We showed that the use of data binned around the missing time interval provides a better imputation than the use of a whole dataset. Imputation was performed for 15 min and 1 h of continuous missing data. We used a dataset with different bin sizes, such as 15 min, 30 min, 45 min, and 1 h, and we carried out evaluations using root mean square error (RMSE) values. We observed that the expectation maximization algorithm worked best for the use of binned data. This was followed by the simple imputer, iterative imputer, and k-nearest neighbor, whereas the random forest method had no effect on data binning during imputation. Moreover, the smallest bin sizes of 15 min and 1 h were observed to provide the lowest RMSE values for the majority of the time frames during the imputation of 15 min and 1 h of missing data, respectively. Although applicable to digital health data, we think that this method will also find applicability in other domains.

Statistical Analysis of the Impact of COVID-19 on PM2.5 Concentrations in Downtown Quito during the Lockdowns in 2020.

In this paper, a comparative analysis between the PM2.5 concentration in downtown Quito, Ecuador, during the COVID-19 pandemic in 2020 and the previous five years (from 2015 to 2019) was carried out. Here, in order to fill in the missing data and achieve homogeneity, eight datasets were constructed, and 35 different estimates were used together with six interpolation methods to put in the estimated value of the missing data. Additionally, the quality of the estimations was verified by using the sum of squared residuals and the following correlation coefficients: Pearson's r, Kendall's τ, and Spearman's ρ. Next, feature vectors were constructed from the data under study using the wavelet transform, and the differences between feature vectors were studied by using principal component analysis and multidimensional scaling. Finally, a robust method to impute missing data in time series and characterize objects is presented. This method was used to support the hypothesis that there were significant differences between the PM2.5 concentration in downtown Quito in 2020 and 2015-2019.

Uncertain imputation for time-series forecasting: Application to COVID-19 daily mortality prediction.

The object of this study is to put forward uncertainty modeling associated with missing time series data imputation in a predictive context. We propose three imputation methods associated with uncertainty modeling. These methods are evaluated on a COVID-19 dataset out of which some values have been randomly removed. The dataset contains the numbers of daily COVID-19 confirmed diagnoses ("new cases") and daily deaths ("new deaths") recorded since the start of the pandemic up to July 2021. The considered task is to predict the number of new deaths 7 days in advance. The more values are missing, the higher the imputation impact is on the predictive performances. The Evidential K-Nearest Neighbors (EKNN) algorithm is used for its ability to take into account labels uncertainty. Experiments are provided to measure the benefits of the label uncertainty models. Results show the positive impact of uncertainty models on imputation performances, especially in a noisy context where the number of missing values is high.

The Impact of Missing Data and Imputation Methods on the Analysis of 24-Hour Activity Patterns.

The purpose of this study is to characterize the impact of the timing and duration of missing actigraphy data on interdaily stability (IS) and intradaily variability (IV) calculation. The performance of three missing data imputation methods (linear interpolation, mean time of day (ToD), and median ToD imputation) for estimating IV and IS was also tested. Week-long actigraphy records with no non-wear or missing timeseries data were masked with zeros or ‘Not a Number’ (NaN) across a range of timings and durations for single and multiple missing data bouts. IV and IS were calculated for true, masked, and imputed (i.e., linear interpolation, mean ToD and, median ToD imputation) timeseries data and used to generate Bland–Alman plots for each condition. Heatmaps were used to analyze the impact of timings and durations of and between bouts. Simulated missing data produced deviations in IV and IS for longer durations, midday crossings, and during similar timing on consecutive days. Median ToD imputation produced the least deviation among the imputation methods. Median ToD imputation is recommended to recapitulate IV and IS under missing data conditions for less than 24 h.

Natural climate reconstruction in the Norwegian stave churches through time series processing with variational autoencoders

PurposeIn the field of heritage science, especially applied to buildings and artefacts made by organic hygroscopic materials, analyzing the microclimate has always been of extreme importance. In particular, in many cases, the knowledge of the outdoor/indoor microclimate may support the decision process in conservation and preservation matters of historic buildings. This knowledge is often gained by implementing long and time-consuming monitoring campaigns that allow collecting atmospheric and climatic data.Design/methodology/approachSometimes the collected time series may be corrupted, incomplete and/or subjected to the sensors' errors because of the remoteness of the historic building location, the natural aging of the sensor or the lack of a continuous check of the data downloading process. For this reason, in this work, an innovative approach about reconstructing the indoor microclimate into heritage buildings, just knowing the outdoor one, is proposed. This methodology is based on using machine learning tools known as variational auto encoders (VAEs), that are able to reconstruct time series and/or to fill data gaps.FindingsThe proposed approach is implemented using data collected in Ringebu Stave Church, a Norwegian medieval wooden heritage building. Reconstructing a realistic time series, for the vast majority of the year period, of the natural internal climate of the Church has been successfully implemented.Originality/valueThe novelty of this work is discussed in the framework of the existing literature. The work explores the potentials of machine learning tools compared to traditional ones, providing a method that is able to reliably fill missing data in time series.

2D and 3D image reconstruction from slice data based on a constrained bilateral smoothing and dynamic mode decomposition

In this paper, we present an algebraic, graph theoretic and data-based image inpainting algorithm. The algorithm is applied to reconstruct area or volume from one-dimensional and two-dimensional slice data. Given one-dimensional or two-dimensional slice data, our algorithm first performs a simple algebraic presmoothing of the data, e.g., Gaussian filters. The second step is to obtain, from these smoothed data, an operator that reconstructs them via Dynamic Mode Decomposition. Dynamic Mode Decomposition is an equation-free modeler, which is designed to construct the discrete dynamical operator considering slice data as a time series data. We interpret the DMD as a solution to the inverse problem and apply an appropriate continuous extension of DMD to fill the missing time series data. The last step is to smooth the outcome from the second step using a Constraint Bilateral Smoother. The Constraint Bilateral Smoother is designed to smooth the boundary of area or volume, respecting the given slice data. We demonstrate the performance of the proposed algorithm, in several test cases, which include an Magnetic Resonance Imaging of a two-year-old and a Computed Tomography scan of a Covid-19 patient. We also show that a couple of test results are better than those from some competitive alternative techniques up to 50%. Our tests also show that the inpainting computation time increases linearly as a function of the number of pixels or voxels with slope just around 0.5.

Predicting in multivariate incomplete time series. Application of the expectation-maximisation algorithm supplemented by the Newton-Raphson method

Statistical practice requires various imperfections resulting from the nature of data to be addressed. Data containing different types of measurement errors and irregularities, such as missing observations, have to be modelled. The study presented in the paper concerns the application of the expectation-maximisation (EM) algorithm to calculate maximum likelihood estimates, using an autoregressive model as an example. The model allows describing a process observed only through measurements with certain level of precision and through more than one data series. The studied series are affected by a measurement error and interrupted in some time periods, which causes the information for parameters estimation and later for prediction to be less precise. The presented technique aims to compensate for missing data in time series. The missing data appear in the form of breaks in the source of the signal. The adjustment has been performed by the EM algorithm to a hybrid version, supplemented by the Newton-Raphson method. This technique allows the estimation of more complex models. The formulation of the substantive model of an autoregressive process affected by noise is outlined, as well as the adjustment introduced to overcome the issue of missing data. The extended version of the algorithm has been verified using sampled data from a model serving as an example for the examined process. The verification demonstrated that the joint EM and Newton-Raphson algorithms converged with a relatively small number of iterations and resulted in the restoration of the information lost due to missing data, providing more accurate predictions than the original algorithm. The study also features an example of the application of the supplemented algorithm to some empirical data (in the calculation of a forecasted demand for newspapers).

A dual-head attention model for time series data imputation

Digital agriculture increasingly relies on the availability and accuracy of measurement data collected from various sensors. Of this data, water quality attracts great attention due to its intended use for crop irrigation, livestock, and other farming activities. Accurate and reliable water quality measurements enable farmers to understand the landscape comprehensively, optimising resource utilisation and reducing the negative impacts of agriculture on the environment. In practice, missing and incomplete data can create biased estimations and reduce the efficiency of many of the valuable applications provided by digital agriculture. The purpose of this paper is to propose a dual-head sequence-to-sequence imputation model (Dual-SSIM) designed to impute missing time series data in sensor networks, therefore reducing the negative consequences of missing and incomplete data. Unlike standard sequence-to-sequence architecture, the Dual-SSIM model features two encoders with gated recurrent units (GRUs) which are used to process temporal information before and after the missing gap separately. Furthermore, the attention mechanism is applied to two encoder outputs concurrently, in order to allow the model to focus on the high relative inputs when estimating missing data. The performance efficacy of Dual-SSIM has been investigated through the monitoring of water quality, sourced from an Australian water quality information system. Experimental results of this investigation indicate that Dual-SSIM outperforms associated alternatives based on the mean absolute error (MAE), root mean square error (RMSE), and dynamic time warping (DTW) scores in imputing two different water quality variables. Therefore, it can be concluded that Dual-SSIM provides an effective and promising approach for water quality data imputation.

Kalman Filter Algorithm versus Other Methods of Estimating Missing Values: Time Series Evidence

Ideally, we think data are carefully collected and have regular patterns with no missing values, but in reality, this does not always happen. This study examines four (4) methods—mean imputation (MI), median imputation (MDI), linear imputation (LI) and Kalman filter algorithm (KAL)—of estimating missing values in time series. The study utilized pairs of nine (9) simulated series; each pair constitutes “actual series” and “12% missingness series”. The three (3) sample sizes i.e. small (50), medium (200) and large (1000) were varied over the additive models linear, quadratic and exponential forms of trend. The 12% missingness series were estimated using MI, MDI, LI and KAL. The performances of the method were checked using the root mean square error (RMSE), mean absolute error (MAE) and mean absolute percentage error (MAPE), while the overall performances of the estimating methods were accessed using the average of the accuracy measures (RMSE, MAE and MAPE). The results of the average-accuracy measures show that KAL outperformed other methods (MI, MDI and LI) at the three sample sizes when the trend was linear; also, MDI outperformed other methods at the three (3) sample sizes when the trend was exponential. Furthermore, MI outperformed others at small and large sample sizes when the trend was quadratic. However, the Kalman filter algorithm proved better when the sample size was medium. Hence, KAL, MI and MDI methods are recommended to estimate missing data in time series when the trend is linear, quadratic and exponential respectively, until further study proves otherwise.

Ensemble Methods for Jump-Diffusion Models of Power Prices

We propose a machine learning-based methodology which makes use of ensemble methods with the aims (i) of treating missing data in time series with irregular observation times and detecting anomalies in the observed time behavior; (ii) of defining suitable models of the system dynamics. We applied this methodology to US wholesale electricity price time series that are characterized by missing data, high and stochastic volatility, jumps and pronounced spikes. For missing data, we provide a repair approach based on the missForest algorithm, an imputation algorithm which is completely agnostic about the data distribution. To identify anomalies, i.e., turbulent movements of power prices in which jumps and spikes are observed, we took into account the no-gap reconstructed electricity price time series, and then we detected anomalous regions using the isolation forest algorithm, an anomaly detection method that isolates anomalies instead of profiling normal data points as in the most common techniques. After removing anomalies, the additional gaps will be newly filled by the missForest imputation algorithm. In this way, a complete and clean time series describing the stable dynamics of power prices can be obtained. The decoupling between the stable motion and the turbulent motion allows us to define suitable jump-diffusion models of power prices and to provide an estimation procedure that uses the full information contained in both the stable and the turbulent dynamics.