Gap-filling Method Research Articles

Spatial Gap-Filling of GK2A Daily Sea Surface Temperature (SST) around the Korean Peninsula Using Meteorological Data and Regression Residual Kriging (RRK)

Satellite remote sensing can measure large ocean surface areas, but the infrared-based sea surface temperature (SST) might not be correctly calculated for the pixels under clouds, resulting in missing values in satellite images. Early studies for the gap-free raster maps of satellite SST were based on spatial interpolation using in situ measurements. In this paper, however, an alternative spatial gap-filling method using regression residual kriging (RRK) for the Geostationary Korea Multi-Purpose Satellite-2A (GK2A) daily SST was examined for the seas around the Korean Peninsula. Extreme outliers were first removed from the in situ measurements and the GK2A daily SST images using multi-step statistical procedures. For the pixels on the in situ measurements after the quality control, a multiple linear regression (MLR) model was built using the selected meteorological variables such as daily SST climatology value, specific humidity, and maximum wind speed. The irregular point residuals from the MLR model were transformed into a residual grid by optimized kriging for the residual compensation for the MLR estimation of the null pixels. The RRK residual compensation method improved accuracy considerably compared with the in situ measurements. The gap-filled 18,876 pixels showed the mean bias error (MBE) of −0.001 °C, the mean absolute error (MAE) of 0.315 °C, the root mean square error (RMSE) of 0.550 °C, and the correlation coefficient (CC) of 0.994. The case studies made sure that the gap-filled SST with RRK had very similar values to the in situ measurements to those of the MLR-only method. This was more apparent in the typhoon case: our RRK result was also stable under the influence of typhoons because it can cope with the abrupt changes in marine meteorology.

Evaluation of gap-filling methods for CH4 flux data based on eddy covariance method in the Lake Taihu, China

Eddy covariance method has become a key technique to measure CH4 flux continuously in lakes. A large number of CH4 flux data was missing due to variable reasons. In order to reconstruct a complete time series of CH4 flux, it is necessary to find an appropriate gap-filling method to insert the CH4 flux data gap. Based on the routine meteorological data and CH4 flux data measured at Bifenggang site in the eastern part of the Taihu eddy flux network during 2014 to 2017, we analyzed the control factors of CH4 flux at the half-hour scale and daily scale. With those data, we tested that whether nonlinear regression method and two machine learning methods, random forest algorithm and error back propagation algorithm, could fill the CH4 flux gap at the half-hour scale and daily scale. The results showed that CH4 flux at the half-hour scale was mainly influenced by sediment temperature, friction velocity, air temperature, relative humidity, latent heat flux and water temperature at 20 cm in the growing season, and was mainly affected by relative humidity, latent heat flux, wind speed, sensible heat flux and sediment temperature in non-growing season. The CH4 flux at the daily scale was mainly affected by latent heat flux and relative humidity. Random forest model was the best in CH4 flux data gap filling at both time scales. The random forest model with the input variables of day of year, solar elevation angle, sediment temperature, friction velocity, air temperature, water temperature at 20 cm, relative humidity, air pressure, and wind speed was more suitable for filling the CH4 flux data gap at the half-hour scale. The random forest model with the input variables of day of year, sediment temperature, friction velocity, air temperature, water temperature at 20 cm, relative humidity, air pressure, wind speed, and downward shortwave radiation was more suitable for filling CH4 flux data gap at the day scale. The interpolation models could fill the data gap better at daily scale than that at the half-hour scale.

Unrootedness: a New Surface in the Lacanian Black Split Subject

How is the split black subject symbolized? The article attempts this question with its Lacanian-styled gap-filling (or -opening) method, using two clues — one from The Big Bang Theory sitcom, and the other from Achebe’s Things Fall Apart. The split subject comes before the imaginary, the mirror stage, which begins with the child’s experience of the Parental Other. Colonization and civil rights then become a later stage in the growth process of the black race, where the white acted as a Parental or Significant Other. Amy Farrell Fowler and Sheldon Cooper from CBS’ The Big Bang Theory (Season 6, Episode 21, first aired on April 25, 2013) both symbolically portray the colonial condition of Africa. After several failed attempts to attain gratification from his Significant Other, Cooper’s revolutionary attention to his “split subject’s needs” without Amy offers a new lens to theorize colonization as a “halfway error” to be corrected and transcended to reach where Okonkwo, in Things Fall Apart, fails. The unrootedness of being black would then polish the new surface to which the entire black race globally can behold and do something not done before.

Effects of Missing Data on Heart Rate Variability Metrics.

Heart rate variability (HRV) has been studied for decades in clinical environments. Currently, the exponential growth of wearable devices in health monitoring is leading to new challenges that need to be solved. These devices have relatively poor signal quality and are affected by numerous motion artifacts, with data loss being the main stumbling block for their use in HRV analysis. In the present paper, it is shown how data loss affects HRV metrics in the time domain and frequency domain and Poincaré plots. A gap-filling method is proposed and compared to other existing approaches to alleviate these effects, both with simulated (16 subjects) and real (20 subjects) missing data. Two different data loss scenarios have been simulated: (i) scattered missing beats, related to a low signal to noise ratio; and (ii) bursts of missing beats, with the most common due to motion artifacts. In addition, a real database of photoplethysmography-derived pulse detection series provided by Apple Watch during a protocol including relax and stress stages is analyzed. The best correction method and maximum acceptable missing beats are given. Results suggest that correction without gap filling is the best option for the standard deviation of the normal-to-normal intervals (SDNN), root mean square of successive differences (RMSSD) and Poincaré plot metrics in datasets with bursts of missing beats predominance (), whereas they benefit from gap-filling approaches in the case of scattered missing beats (). Gap-filling approaches are also the best for frequency-domain metrics (). The findings of this work are useful for the design of robust HRV applications depending on missing data tolerance and the desired HRV metrics.

A physical full-factorial scheme for gap-filling of eddy covariance measurements of daytime evapotranspiration

Surface evapotranspiration (ET) is a vital process that connects the water cycle, energy budget, and carbon cycle between the land and atmosphere. ET measurements using the eddy covariance (EC) technique typically encounter large gaps. In this study, a physics-based full-factorial scheme for filling gaps in ET from EC observations is proposed based on the decoupling model of the Penman-Monteith equation, which mechanically interconnects the full range of ET influential factors from the atmosphere, vegetation, and soil. The new method was tested and intercompared with four typical gap-filling methods, i.e., marginal distribution sampling, mean diurnal variation, constant reference evaporative fraction, and constant evaporative fraction, using the data collected at 136 sites from AmeriFlux, FLUXNET, and Tibetan Plateau Data Center over the crop, grass, forest, and other remaining land-cover types. The validation results showed that (1) the full-factorial scheme performed well for filling the artificially randomly generated hourly and daily gaps of the EC-based ET measurements, with a root mean square error (RMSE) of 28.8–96.1 W/m2 for the hourly gaps and an RMSE of 19.9–38.7 W/m2 for the daily gaps, (2) for filling the hourly gaps, the accuracy decreased with the increase in the gap length; for filling the daily gaps, the accuracy decreased with the increase in the gap length at first and then remained almost unchanged, (3) the full-factorial scheme outperformed the four typical methods for filling both the hourly and daily gaps, and (4) all five gap-filling methods presented the best performance at the grass sites, followed by the crop, and the worst performance at the forest sites. In conclusion, the proposed full-factorial scheme can produce reasonable gap-filled hourly and daily ET and is superior to the existing typical gap-filling methods. Therefore, it could be a good candidate for filling the ET gaps in EC measurements.

Does the gap-filling method influence long-term (1950–2019) temperature and precipitation trend analyses?

Does the gap-filling method influence long-term (1950–2019) temperature and precipitation trend analyses?

Impact of Preprocessing on Tree Canopy Cover Modelling: Does Gap-Filling of Landsat Time Series Improve Modelling Accuracy?

Preprocessing of Landsat images is a double-edged sword, transforming the raw data into a useful format but potentially introducing unwanted values with unnecessary steps. Through recovering missing data of satellite images in time series analysis, gap-filling is an important, highly developed, preprocessing procedure, but its necessity and effects in numerous Landsat applications, such as tree canopy cover (TCC) modelling, are rarely examined. We address this barrier by providing a quantitative comparison of TCC modelling using predictor variables derived from Landsat time series that included gap-filling versus those that did not include gap-filling and evaluating the effects that gap-filling has on modelling TCC. With 1-year Landsat time series from a tropical region located in Taita Hills, Kenya, and a reference TCC map in 0–100 scales derived from airborne laser scanning data, we designed comparable random forest modelling experiments to address the following questions: 1) Does gap-filling improve TCC modelling based on time series predictor variables including the seasonal composites (SC), spectral-temporal metrics (STMs), and harmonic regression (HR) coefficients? 2) What is the difference in TCC modelling between using gap-filled pixels and using valid (actual or cloud-free) pixels? Two gap-filling methods, one temporal-based method (Steffen spline interpolation) and one hybrid method (MOPSTM) have been examined. We show that gap-filled predictors derived from the Landsat time series delivered better performance on average than non-gap-filled predictors with the average of median RMSE values for Steffen-filled and MOPSTM-filled SC’s being 17.09 and 16.57 respectively, while for non-gap-filled predictors, it was 17.21. MOPSTM-filled SC is 3.7% better than non-gap-filled SC on RMSE, and Steffen-filled SC is 0.7% better than non-gap-filled SC on RMSE. The positive effects of gap-filling may be reduced when there are sufficient high-quality valid observations to generate a seasonal composite. The single-date experiment suggests that gap-filled data (e.g. RMSE of 16.99, 17.71, 16.24, and 17.85 with 100% gap-filled pixels as training and test datasets for four seasons) may deliver no worse performance than valid data (e.g. RMSE of 15.46, 17.07, 16.31, and 18.14 with 100% valid pixels as training and test datasets for four seasons). Thus, we conclude that gap-filling has a positive effect on the accuracy of TCC modelling, which justifies its inclusion in image preprocessing workflows.

Estimating Full-Coverage PM2.5 Concentrations Based on Himawari-8 and NAQPMS Data over Sichuan-Chongqing

Fine particulate matter (PM2.5) has attracted extensive attention due to its harmful effects on humans and the environment. The sparse ground-based air monitoring stations limit their application for scientific research, while aerosol optical depth (AOD) by remote sensing satellite technology retrieval can reflect air quality on a large scale and thus compensate for the shortcomings of ground-based measurements. In this study, the elaborate vertical-humidity method was used to estimate PM2.5 with the spatial resolution 1 km and the temporal resolution 1 hour. For vertical correction, the scale height of aerosols (Ha) was introduced based on the relationship between the visibility data and extinction coefficient of meteorological observations to correct the AOD of the Advance Himawari Imager (AHI) onboard the Himawari-8 satellite. The hygroscopic growth factor (f(RH)) was fitted site-by-site and month by month (1–12 months). Meanwhile, the spatial distribution of the fitted coefficients can be obtained by interpolation assuming that the aerosol properties vary smoothly on a regional scale. The inverse distance weighted (IDW) method was performed to construct the hygroscopic correction factor grid for humidity correction so as to estimate the PM2.5 concentrations in Sichuan and Chongqing from 09:00 to 16:00 in 2017–2018. The results indicate that the correlation between “dry” extinction coefficient and PM2.5 is slightly improved compared to the correlation between AOD and PM2.5, with r coefficient values increasing from 0.12–0.45 to 0.32–0.69. The r of hour-by-hour verification is between 0.69 and 0.85, and the accuracy of the afternoon is higher than that of the morning. Due to the missing rate of AOD in the southwest is very high, this study utilized inverse variance weighting (IVW) gap-filling method combine satellite estimation PM2.5 and the nested air-quality prediction modeling system (NAQPMS) simulation data to obtain the full-coverage hourly PM2.5 concentration and analyze a pollution process in the fall and winter.

The telomere-to-telomere gap-free genome of four rice parents reveals SV and PAV patterns in hybrid rice breeding.

The telomere-to-telomere gap-free genome of four rice parents reveals SV and PAV patterns in hybrid rice breeding.

Dynamic mass loss from Greenland's marine-terminating peripheral glaciers (1985–2018)

AbstractGlobal glacier mass balance decreased rapidly over the last two decades, exceeding mass loss from the Greenland and Antarctic Ice Sheets. In Greenland, peripheral glaciers and ice caps (GICs) cover only ~5% of Greenland's area but contributed ~20% of the island's ice mass loss between 2000 and 2018. Although Greenland GIC mass loss due to surface meltwater runoff has been estimated using atmospheric models, mass lost to changes in ice discharge into oceans (i.e., dynamic mass loss) remains unquantified. We use the flux gate method to estimate discharge from Greenland's 585 marine-terminating peripheral glaciers between 1985 and 2018, and compute dynamic mass loss as the discharge anomaly relative to the 1985–98 period. Greenland GICs discharged between 2.94 ± 0.23 and 4.03 ± 0.23 Gt a−1from 1985 to 1998, depending on the gap-filling method, and abruptly increased to 5.10 ± 0.21 Gt a−1from 1999 to 2018. The resultant ~1–2 Gt a−1dynamic mass loss was driven by synchronous widespread acceleration around Greenland. The mass loss came predominantly from the southeast region, which contains 39% of the glaciers. Although changes in discharge over time were small relative to surface mass-balance changes, our speed and discharge time series suggest these glaciers may quickly accelerate in response to changes in climate.

A new object-class based gap-filling method for PlanetScope satellite image time series

PlanetScope CubeSats data with a 3-m resolution, frequent revisits, and global coverage have provided an unprecedented opportunity to advance land surface monitoring over the recent years. Similar to other optical satellites, cloud-induced data missing in PlanetScope satellites substantially hinders its use for broad applications. However, effective gap-filling in PlanetScope image time series remains challenging and is subject to whether it can 1) consistently generate high accuracy results regardless of different gap sizes, especially for heterogeneous landscapes, and 2) effectively recover the missing pixels associated with rapid land cover changes. To address these challenges, we proposed an object-class based gap-filling (‘OCBGF’) method. Two major novelties of OCBGF include 1) adopting an object-based segmentation method in conjunction with an unsupervised classification method to help characterize the landscape heterogeneity and facilitate the search of neighboring valid pixels for gap-filling, improving its applicability regardless of the gap size; 2) employing a scenario-specific gap-filling approach that enables effective gap-filling of areas with rapid land cover change. We tested OCBGF at four sites representative of different land cover types (plantation, cropland, urban, and forest). For each site, we evaluated the performance of OCBGF on both simulated and real cloud-contaminated scenarios, and compared our results with three state-of-the-art methods, namely Neighborhood Similar Pixel Interpolator (NSPI), AutoRegression to Remove Clouds (ARRC), and Spectral-Angle-Mapper Based Spatio-Temporal Similarity (SAMSTS). Our results show that across all four sites, OCBGF consistently obtains the highest accuracy in gap-filling when applied to scenarios with various gap sizes (RMSE = 0.0065, 0.0090, 0.0092, and 0.0113 for OCBGF, SAMSTS, ARRC, and NSPI, respectively) and with/without rapid land cover changes (RMSE = 0.0082, 0.0112, 0.0119, and 0.0120 for OCBGF, SAMSTS, ARRC, and NSPI, respectively). These results demonstrate the effectiveness of OCBGF for gap-filling PlanetScope image time series, with potential to be extended to other satellites.

CLIMFILL v0.9: a framework for intelligently gap filling Earth observations

Abstract. Remotely sensed Earth observations have many missing values. The abundance and often complex patterns of these missing values can be a barrier for combining different observational datasets and may cause biased estimates of derived statistics. To overcome this, missing values in geoscientific data are regularly infilled with estimates through univariate gap-filling techniques such as spatial or temporal interpolation or by upscaling approaches in which complete donor variables are used to infer missing values. However, these approaches typically do not account for information that may be present in other observed variables that also have missing values. Here we propose CLIMFILL (CLIMate data gap-FILL), a multivariate gap-filling procedure that combines kriging interpolation with a statistical gap-filling method designed to account for the dependence across multiple gappy variables. In a first stage, an initial gap fill is constructed for each variable separately using state-of-the-art spatial interpolation. Subsequently, the initial gap fill for each variable is updated to recover the dependence across variables using an iterative procedure. Estimates for missing values are thus informed by knowledge of neighbouring observations, temporal processes, and dependent observations of other relevant variables. CLIMFILL is tested using gap-free ERA-5 reanalysis data of ground temperature, surface-layer soil moisture, precipitation, and terrestrial water storage to represent central interactions between soil moisture and climate. These variables were matched with corresponding remote sensing observations and masked where the observations have missing values. In this “perfect dataset approach” CLIMFILL can be evaluated against the original, usually not observed part of the data. We show that CLIMFILL successfully recovers the dependence structure among the variables across all land cover types and altitudes, thereby enabling subsequent mechanistic interpretations in the gap-filled dataset. Correlation between original ERA-5 data and gap-filled ERA-5 data is high in many regions, although it shows artefacts of the interpolation procedure in large gaps in high-latitude regions during winter. Bias and noise in gappy satellite-observable data is reduced in most regions. A case study of the European 2003 heatwave shows how CLIMFILL reduces biases in ground temperature and surface-layer soil moisture induced by the missing values. Furthermore, in idealized experiments we see the impact of fraction of missing values and the complexity of missing value patterns to the performance of CLIMFILL, showing that CLIMFILL for most variables operates at the upper limit of what is possible given the high fraction of missing values and the complexity of missingness patterns. Thus, the framework can be a tool for gap filling a large range of remote sensing observations commonly used in climate and environmental research.

Estimation of Landsat-like daily evapotranspiration for crop water consumption monitoring using TSEB model and data fusion.

Evapotranspiration (ET) plays an essential role in agricultural water resource management. Understanding regional agricultural water consumption characteristics can be improved by predicting ET using remote sensing. However, due to the lack of high-resolution images on clear-sky days or the limitation of ET reconstruction on cloudy-sky days, it remains challenging to continuously derive ET at the field scale. In this study, the Landsat and MODIS data were initially fused to obtain the Landsat-like vegetation index and land surface temperature on clear-sky days. Then the two-source energy balance (TSEB) model was applied to calculate the daily ET during the clear-sky. A canopy resistance-based gap-filling method was involved in reconstructing regional ET on cloudy days while considering different environmental factors. The estimations were validated by automatic weather system data (AWS) and eddy covariance (EC) measurements in Guantao County. The results demonstrated that the proposed scheme performed well in estimating cropland ET, with an RMSE of 0.86 mm·d−1 and an R2 of 0.65, and the NSE and PBias were 0.61 and -0.29%, respectively. The crop water consumption analysis revealed that the daily ET of winter wheat peaked during the maturation stage. Nevertheless, summer maize water consumption peaked in the middle of the growing season in this area. The temperature during the early development stage and the soil moisture in the mid and late growth stages had the greatest impact on the ET of winter wheat. During the entire growing period, soil moisture had the largest effect on the ET of summer maize. The findings showed that the TSEB model can be effectively applied to field-scale water consumption monitoring in North China through MODIS and Landsat data fusion and ET temporal reconstruction considering environmental factors.

Regression-based gap-filling methods show air temperature reductions and wind pattern changes during the 2019 total eclipse in Chile

Singular disruptive events like solar eclipses affect the measured values of meteorological variables at the earth’s surface. To quantify such an impact, it is necessary to estimate what value the parameter would have taken had the event not occurred. We design and compare several methods to perform such an estimate based on longer observational timeseries from individual meteorological surface stations. Our methods are based on regularised regressions (including a Bayesian variant) and provide both a point an associated error estimate of the disruptive event’s impact. With their help, we study the effect of the total solar eclipse of July 2nd, 2019, in the Coquimbo Region of Chile, on near-surface air temperatures and winds. The observational data used have been collected by the meteorological surface station network of the Centro de Estudios Avanzados en Zonas Áridas (CEAZA). Most stations inside the eclipse’s umbra registered a temperature drop of 1–2 ^{circ }C, while the most extreme estimated temperature drop surpassed 6 ^{circ }C. The presence of an ‘eclipse cyclone’ can neither be proven nor refuted. Application of the regression methods to other comparable problems like volcanic eruptions, forest fires, or simply gap filling of observational data, are conceivable.

Annual CO2 Budget Estimation From Chamber-Based Flux Measurements on Intensively Drained Peat Meadows: Effect of Gap-Filling Strategies

Estimating annual CO2 budgets on drained peatlands is important in understanding the significance of CO2 emissions from peatland degradation and evaluating the effectiveness of mitigation techniques. The closed-chamber technique is widely used in combination with gap-filling of CO2 fluxes by parameter fitting empirical models of ecosystem respiration (Reco) and gross primary production (GPP). However, numerous gap-filling strategies are available which are suitable for different circumstances and can result in large variances in annual budget estimates. Therefore, a need for guidance on the selection of gap-filling methodology and its influence on the results exists. Here, we propose a framework of gap-filling methods with four Tiers following increasing model complexity at structural and temporal levels. Tier one is a simple parameter fitting of basic empirical models on an annual basis. Tier two adds structural complexity by including extra environmental factors such as grass height, groundwater level and drought condition. Tier three introduces temporal complexity by separation of annual datasets into seasons. Tier four is a campaign-specific parameter fitting approach, representing highest temporal complexity. The methods were demonstrated on two chamber-based CO2 flux datasets, one of which was previously published. Performance of the empirical models were compared in terms of error statistics. Annual budget estimates were indirectly validated with carbon export values. In conclusion, different gap-filling methodologies gave similar annual estimates but different intra-annual CO2 fluxes, which did not affect the detection of the treatment effects. The campaign-wise gap-filling at Tier four gave the best model performances, while Tier three seasonal gap-filling produced satisfactory results throughout, even under data scarcity. Given the need for more complete carbon balances in drained peatlands, our four-Tier framework can serve as a methodological guidance to the handling of chamber-measured CO2 fluxes, which is fundamental in understanding emissions from degraded peatlands and its mitigation. The performance of models on intra-annual data should be validated in future research with continuous measured CO2 flux data.

Comparative characteristics of gap filling methods in high-frequency data of micrometeorological measurements

This article is devoted to one of the most complex technical aspects of the primary processing of micrometeorological data. The article contains a description of basic algorithms for restoring missing values in time series of data. The gap filling methods implemented by the authors on the basis of the thus described algorithms are tested using an example of filling-in generated gaps in series of temperature and vertical wind speed. The series of high-frequency measurements of an acoustic anemometer installed on the micrometeorological tower of the meteorological observatory (MO) of Moscow State University (MSU) are used as test data. As a result of this review of gap-filling methods, the effectiveness of various methods for filling-in missing values is assessed.

Gap-filling MODIS daily aerosol optical depth products by developing a spatiotemporal fitting algorithm

ABSTRACT Aerosol loadings and their spatial distribution are among the most important atmospheric information needed for a range of applications such as air quality monitoring, climate research, and public health. A key measure of aerosol quantity is aerosol optical depth (AOD) and it has been routinely observed from space by Earth observing satellites/instrument, especially the Moderate Resolution Imaging Spectroradiometer (MODIS). Despite its global coverage and daily temporal resolution, the MODIS Multi-Angle Implementation of Atmospheric Correction (MAIAC) AOD product is fraught with missing values, severely limiting its use. A gap-filling method which is suitable for large-area application with high efficiency to obtain gapless AOD with reasonable spatial pattern and complete coverage is still lacking. Here, we proposed a novel spatiotemporal fitting algorithm to gap-fill the daily MODIS AOD product. Our algorithm is a multi-stage method aimed to address the non-stationary nature of AOD time series. First, the trend of daily AOD in a year in each pixel was fitted via smoothing splines and the residual was derived based on the original data and the trend. Second, the residual was spatially interpolated, leveraging the spatial correlation between the target pixel and the neighboring pixels. Third, the actual AOD was calculated as the sum of the trend and interpolated residual. We tested the algorithm against ground-based AOD data from 2011 to 2018 in China and further evaluated it via cross-validation at the global scale based on 10 selected MODIS tiles. Compared to the ground-reference AOD, the RMSE of our gapless datasets were 0.24 and 0.27 for Terra and Aqua, respectively; and the cross-validation showed a RMSE ranging from 0.045 to 0.055 (Terra) and 0.047 to 0.057 (Aqua) under different missing ratios. The novel gap-filling method outperforms the Interpolation-based Correlation Weighting (ICW) and Inverse Distance Weighting (IDW) algorithms in accuracy. Meanwhile, the gapless AOD using the novel algorithm shows lower accuracy than original MAIAC AOD, similar accuracy with the AOD from the Long-term Gap-free High-resolution Air Pollutants (LGHAP) concentration dataset, higher accuracy than the AOD from the Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA-2). Overall, the accuracy of gapless AOD using this algorithm meets the need of typical applications in relevant studies. The proposed algorithm is transferable to other regions, with the potential to be used even operationally and efficiently for generating accurate gapless global daily AOD datasets with the input of only MODIS MAIAC AOD data.

Technical note: Using long short-term memory models to fill data gaps in hydrological monitoring networks

Abstract. Quantifying the spatiotemporal dynamics in subsurface hydrological flows over a long time window usually employs a network of monitoring wells. However, such observations are often spatially sparse with potential temporal gaps due to poor quality or instrument failure. In this study, we explore the ability of recurrent neural networks to fill gaps in a spatially distributed time-series dataset. We use a well network that monitors the dynamic and heterogeneous hydrologic exchanges between the Columbia River and its adjacent groundwater aquifer at the U.S. Department of Energy's Hanford site. This 10-year-long dataset contains hourly temperature, specific conductance, and groundwater table elevation measurements from 42 wells with gaps of various lengths. We employ a long short-term memory (LSTM) model to capture the temporal variations in the observed system behaviors needed for gap filling. The performance of the LSTM-based gap-filling method was evaluated against a traditional autoregressive integrated moving average (ARIMA) method in terms of error statistics and accuracy in capturing the temporal patterns of river corridor wells with various dynamics signatures. Our study demonstrates that the ARIMA models yield better average error statistics, although they tend to have larger errors during time windows with abrupt changes or high-frequency (daily and subdaily) variations. The LSTM-based models excel in capturing both high-frequency and low-frequency (monthly and seasonal) dynamics. However, the inclusion of high-frequency fluctuations may also lead to overly dynamic predictions in time windows that lack such fluctuations. The LSTM can take advantage of the spatial information from neighboring wells to improve the gap-filling accuracy, especially for long gaps in system states that vary at subdaily scales. While LSTM models require substantial training data and have limited extrapolation power beyond the conditions represented in the training data, they afford great flexibility to account for the spatial correlations, temporal correlations, and nonlinearity in data without a priori assumptions. Thus, LSTMs provide effective alternatives to fill in data gaps in spatially distributed time-series observations characterized by multiple dominant frequencies of variability, which are essential for advancing our understanding of dynamic complex systems.

Introducing State-of-the-Art Deep Learning Technique for Gap-Filling of Eddy Covariance Crop Evapotranspiration Data

Gaps often occur in eddy covariance flux measurements, leading to data loss and necessitating accurate gap-filling. Furthermore, gaps in evapotranspiration (ET) measurements of annual field crops are particularly challenging to fill because crops undergo rapid change over a short season. In this study, an innovative deep learning (DL) gap-filling method was tested on a database comprising six datasets from different crops (cotton, tomato, and wheat). For various gap scenarios, the performance of the method was compared with the common gap-filling technique, marginal distribution sampling (MDS), which is based on lookup tables. Furthermore, a predictor importance analysis was performed to evaluate the importance of the different meteorological inputs in estimating ET. On the half-hourly time scale, the deep learning method showed a significant 13.5% decrease in nRMSE (normalized root mean square error) throughout all datasets and gap durations. A substantially smaller standard deviation of mean nRMSE, compared with marginal distribution sampling, was also observed. On the whole-gap time scale (half a day to six days), average nMBE (normalized mean bias error) was similar to that of MDS, whereas standard deviation was improved. Using only air temperature and relative humidity as input variables provided an RMSE that was significantly smaller than that resulting from the MDS method. These results suggest that the deep learning method developed here is reliable and more consistent than the standard gap-filling method and thereby demonstrates the potential of advanced deep learning techniques for improving dynamic time series modeling.

Spectral Temporal Information for Missing Data Reconstruction (STIMDR) of Landsat Reflectance Time Series

The number of Landsat time-series applications has grown substantially because of its approximately 50-year history and relatively high spatial resolution for observing long term changes in the Earth’s surface. However, missing observations (i.e., gaps) caused by clouds and cloud shadows, orbit and sensing geometry, and sensor issues have broadly limited the development of Landsat time-series applications. Due to the large area and temporal and spatial irregularity of time-series gaps, it is difficult to find an efficient and highly precise method to fill them. The Missing Observation Prediction based on Spectral-Temporal Metrics (MOPSTM) method has been proposed and delivered good performance in filling large-area gaps of single-date Landsat images. However, it can be less practical for a time series longer than one year due to the lack of mechanics that exclude dissimilar data in time series (e.g., different phenology or changes in land cover). To solve this problem, this study proposes a new gap-filling method, Spectral Temporal Information for Missing Data Reconstruction (STIMDR), and examines its performance in Landsat reflectance time series. Two groups of experiments, including 2000 × 2000 pixel Landsat single-date images and Landsat time series acquired from four sites (Kenya, Finland, Germany, and China), were performed to test the new method. We simulated artificial gaps to evaluate predicted pixel values with real observations. Quantitative and qualitative evaluations of gap-filled images through comparisons with other state-of-the-art methods confirmed the more robust and accurate performance of the proposed method. In addition, the proposed method was also able to fill gaps contaminated by extreme cloud cover for a period (e.g., winter in high-latitude areas). A down-stream task of random forest supervised classification through both gap-filled simulated datasets and the original valid datasets verified that STIMDR-generated products are relevant to the user community for land cover applications.