Sub-hourly Resolution Research Articles

Ensemble machine learning for interpretable soil heat flux estimation

Soil heat flux (SHF) is a key component of the surface energy balance and driver of soil physiochemical and biological processes. Accurate estimation of soil heat flux is challenging due to variations in soil composition, overlying vegetation density and phenology, and highly variable environmental forcings. Existing SHF process-based and data-driven estimation methods have focused on midday landscape scale estimates that correspond to satellite acquisitions, despite the high variability that SHF displays at diurnal scales and throughout the growing season. Recently developed data-driven techniques have emphasized pre-determined predictor variables, with model complexity and predictor selection not carefully evaluated, leaving a gap in our understanding of what information is required to accurately predict SHF. Here we developed and evaluated a suite of ensemble machine learning (ML) models to quantify the ability of meteorological and remote sensing data to predict SHF variability at half-hourly temporal resolution throughout a growing season, producing a comprehensive evaluation of the importance of predictor set composition for SHF estimation at high temporal resolution. We compared this suite of machine learning models to six semi-empirical models and found that the machine learning models broadly outperformed the existing models in capturing diurnal variability across the growing season for four agro-ecosystems (soybean, corn, sorghum, and miscanthus). Crop-specific ML models were able to capture over 86% of the variability in SHF using only two predictor variables, pointing to the need for careful evaluation of predictor sets to identify synergistic combinations. ML models developed using pooled data across all crops captured almost 80% of SHF variability using three predictors, demonstrating the power and generalizability of these methods independent of crop type. Shapley additive explanations (SHAP) were used to examine model interpretability, providing insights into the typically opaque ML modelling process and interaction of predictor variables. Models trained with fewer input variables tended to display more linear and interpretable feature attribution, motivating the use of interpretability as an important consideration in parsimonious model selection. These results provide a robust demonstration of the ability of ML to capture the variability in SHF at sub-hourly resolution across growing seasons spanning a wide range of phenological variation for unique agricultural systems. This study provides a comprehensive evaluation of predictor requirements for model performance, guiding future applications that will take advantage of the next generation of satellite-based observing systems, or in-situ proximal observations of vegetation status and meteorological conditions.

Reconstructing hourly residential electrical load profiles for Renewable Energy Communities using non-intrusive machine learning techniques

The successful implementation of Renewable Energy Communities (RECs) involves maximizing the self-consumption within a community, particularly in regulatory contexts in which shared energy is incentivized. In many countries, the absence of a metering infrastructure that provides data at an hourly or sub-hourly resolution level for low-voltage users (e.g., residential and commercial users) makes the design of a new energy community a challenging task. This study proposes a non-intrusive machine learning methodology that can be used to generate residential electrical consumption profiles at an hourly resolution level using only monthly consumption data (i.e., billed energy), with the aim of estimating the energy shared by RECs. The proposed methodology involves three phases: first, identifying the typical load patterns of residential users through k-Means clustering, then implementing a Random Forest algorithm, based on monthly energy bills, to identify typical load patterns and, finally, reconstructing the hourly electrical load profile through a data-driven rescaling procedure. The effectiveness of the proposed methodology has been evaluated through an REC case study composed by 37 residential users powered by a 70 kWp photovoltaic plant. The Normalized Mean Absolute Error (NMAE) and the Normalized Root Mean Squared Error (NRMSE) were evaluated over an entire year and whenever the energy was shared within the REC. The Relative Absolute Error was also measured when estimating the shared energy at both a monthly (MRAE) and at an annual basis. (RAE). A comparison between the REC load profile reconstructed using the proposed methodology and the real load profile yielded an overall NMAE of 20.04 %, an NRMSE of 26.17 %, and errors of 18.34 % and 23.87 % during shared energy timeframes, respectively. Furthermore, our model delivered relative absolute errors for the estimation of the shared energy at a monthly and annual scale of 8.31 % and 0.12 %, respectively.

Improving satellite-based global rainfall erosivity estimates through merging with gauge data

Rainfall erosivity is a key factor that influences soil erosion by water. Rain-gauge measurements are commonly used to estimate rainfall erosivity. However, long-term gauge records with sub-hourly resolutions are lacking in large parts of the world. Satellite observations provide spatially continuous estimates of rainfall, but they are subject to biases that affect estimates of rainfall erosivity. We employed a novel approach to map global rainfall erosivity based on a high-temporal-resolution (30-min), long-term (2001–2020) satellite-based precipitation product—the Integrated Multi-satellitE Retrievals for Global Precipitation Measurement (GPM-IMERG)—and mean annual rainfall erosivity from the Global Rainfall Erosivity Database (GloREDa) stations (n = 3286). We used a residual-based merging scheme to integrate GPM-IMERG-based rainfall erosivity with GloREDa using Geographically Weighted Regression (GWR). The accuracy of the GWR-based merging scheme was evaluated with a 10-fold cross-validation against GloREDa stations. Based on GPM-IMERG-only, the global mean annual rainfall erosivity was estimated to be 1173 MJ mm ha−1 h−1 yr−1 with a standard deviation of 1736 MJ mm ha−1 h−1 yr−1. The mean value estimated via GPM-IMERG merged with GloREDa was 2020 MJ mm ha−1 h−1 yr−1 with a standard deviation of 3415 MJ mm ha−1 h−1 yr−1. Overall, GPM-IMERG-only estimates underestimated rainfall erosivity. The underestimations were greatest in areas of high rainfall erosivity. The accuracy of rainfall erosivity estimates from GPM-IMERG merged with GloREDa substantially improved (Nash-Sutcliffe efficiency = 0.83, percent bias = −2.4%, and root mean square error = 1122 MJ mm ha−1 h−1 yr−1) compared to estimates by GPM-IMERG-only (Nash-Sutcliffe efficiency = 0.51, percent bias = 27.8%, and root mean square error = 1730 MJ mm ha−1 h−1 yr−1). Improving satellite-based global rainfall erosivity estimates through integrating with gauge data is relevant as it can contribute to enhancing soil erosion modeling and, in turn, support land degradation neutrality programs.

Sub-hourly resolution quality control of rain-gauge data significantly improves regional sub-daily return level estimates.

This research demonstrates how the use of high-resolution rain-gauge data for quality control (QC) significantly changes extreme rainfall estimates, with implications in scientific, meteorological and engineering applications. Current open QC algorithms only consider data at hourly or daily accumulations. Here we present the first open QC algorithm utilising sub-hourly rain-gauge data from official networks at a national, multi-decade scale. We use data from 1,301 rain-gauges in Great Britain (GB) to develop a threshold-based methodology for sub-hourly QC that can be used to complement existing, freely available hourly QC methods by developing an algorithm for sub-hourly QC that uses monthly thresholds for 1 hr, 15 min and 1 min rainfall totals. We then evaluated the effect of combining these QC procedures on rainfall distributions using graphical and statistical methods, with an emphasis on extreme value analysis. We demonstrate that the additional information in sub-hourly rainfall allows our new QC to remove spuriously large values undetected by existing methods which generate errors in extreme rainfall estimates. This results in statistically significant differences between extreme rainfall estimates for 15 min and 1 hr accumulations, with smaller differences found for 6 and 24 hr totals. We also find that extremes in the distributions of 15 min and 1 hr rainfall accumulations tend to grow more rapidly with return period than for longer accumulation periods. We observe similarities between the shape parameter populations for 15 min and 1 hr rainfall accumulations, suggesting that hourly records may be used to improve shape parameter estimates for extreme sub-hourly rainfall in GB. Sub-hourly QC moderates unrealistically large return level estimates for short-duration rainfall, with beneficial impacts on data required for the design of urban drainage infrastructure and the validation of high-resolution climate models.

Why should we care about high temporal resolution monitoring of bioaerosols in ambient air?

This is the first time that atmospheric concentrations of individual pollen types have been recorded by an automatic sampler with 1-hour and sub-hourly resolution (i.e. 1-minute and 1-second data). The data were collected by traditional Hirst type methods and state-of the art Rapid-E real-time bioaerosol detector. Airborne pollen data from 7 taxa, i.e. Acer negundo, Ambrosia, Broussonetia papyrifera, Cupressales (Taxaceae and Cupressaceae families), Platanus, Salix and Ulmus, were collected during the 2019 pollen season in Novi Sad, Serbia. Pollen data with daily, hourly and sub-hourly temporal resolution were analysed in terms of their temporal variability. The impact of turbulence kinetic energy (TKE) on pollen cloud homogeneity was investigated. Variations in Seasonal Pollen Integrals produced by Hirst and Rapid-E show that scaling factors are required to make data comparable. Daily average and hourly measurements recorded by the Rapid-E and Hirst were highly correlated and so examining Rapid-E measurements with sub-hourly resolution is assumed meaningful from the perspective of identification accuracy. Sub-hourly data provided an insight into the heterogenous nature of pollen in the air, with distinct peaks lasting ~5–10 min, and mostly single pollen grains recorded per second. Short term variations in 1-minute pollen concentrations could not be wholly explained by TKE. The new generation of automatic devices has the potential to increase our understanding of the distribution of bioaerosols in the air, provide insights into biological processes such as pollen release and dispersal mechanisms, and have the potential for us to conduct investigations into dose-response relationships and personal exposure to aeroallergens.

Combination of meteorological reanalysis data and stochastic simulation for modelling wind generation variability

Abstract As installed wind generation capacities increase, there is a need to model variability in wind generation in detail to analyse its impacts on power systems. Utilization of meteorological reanalysis data and stochastic simulation are possible approaches for modelling this variability. In this paper, a combination of these two approaches is used to model wind generation variability. Parameters for the model are determined based on measured wind speed data. The model is used to simulate wind generation from the level of a single offshore wind power plant to the aggregate onshore wind generation of western Denmark. The simulations are compared to two years of generation measurements on 15 min resolution. The results indicate that the model, combining reanalysis data and stochastic simulation, can successfully model wind generation variability on different geographical aggregation levels on sub-hourly resolution. It is shown that the addition of stochastic simulation to reanalysis data is required when modelling offshore wind generation and when analysing onshore wind in small geographical regions.

Imputation of missing sub-hourly precipitation data in a large sensor network: A machine learning approach

Precipitation data collected at sub-hourly resolution represents specific challenges for missing data recovery by being largely stochastic in nature and highly unbalanced in the duration of rain vs non-rain. Here we present a two-step analysis utilising current machine learning techniques for imputing precipitation data sampled at 30-minute intervals by devolving the task into (a) the classification of rain or non-rain samples, and (b) regressing the absolute values of predicted rain samples. Investigating 37 weather stations in the UK, this machine learning process produces more accurate predictions for recovering precipitation data than an established surface fitting technique utilising neighbouring rain gauges. Increasing available features for the training of machine learning algorithms increases performance with the integration of weather data at the target site with externally sourced rain gauges providing the highest performance. This method informs machine learning models by utilising information in concurrently collected environmental data to make accurate predictions of missing rain data. Capturing complex non-linear relationships from weakly correlated variables is critical for data recovery at sub-hourly resolutions. Such pipelines for data recovery can be developed and deployed for highly automated and near instantaneous imputation of missing values in ongoing datasets at high temporal resolutions.

An inverse approach to solving zone air infiltration rate and people count using indoor environmental sensor data

Physics-based simulation of energy use in buildings is widely used in building design and performance rating, controls design and operations. However, various challenges exist in the modeling process. Model parameters such as people count and air infiltration rate are usually highly uncertain, yet they have significant impacts on the simulation accuracy. With the increasing availability and affordability of sensors and meters in buildings, a large amount of measured data has been collected including indoor environmental parameters, such as room air dry-bulb temperature, humidity ratio, and CO2 concentration levels. Fusing these sensor data with traditional energy modeling poses new opportunities to improve simulation accuracy. This study develops a set of physics-based inverse algorithms which can solve the highly uncertain and hard-to-measure building parameters such as zone-level people count and air infiltration rate. A simulation-based case study is conducted to verify the inverse algorithms implemented in EnergyPlus covering various sensor measurement scenarios and different modeling use cases. The developed inverse models can solve the zone people count and air infiltration at sub-hourly resolution using the measured zone air temperature, humidity and/or CO2 concentration given other easy-to-measure model parameters are known.

Flexibility contribution of heat ventilation and air conditioning loads in a multi‐stage robust unit commitment with non‐deterministic variability‐oriented ramp reserves

Aside from conventional generation units, heat ventilation and air-conditioning (HVAC) loads, thanks to their inherent thermal capacity storage, are reasonable alternatives to mitigate short-term variability impacts in power systems with high renewable energy sources (RES). Accordingly, HVAC loads are integrated into a multi-stage multi-resolution robust unit commitment considering non-deterministic variability-oriented reserves to address operational flexibility requirement in power systems. Since common two-stage robust optimisations are over-conservative, non-causal and in general NP-hard, a non-conservative extended affinely adjustable robust optimisation approach is proposed to provide causality, enhance computational tractability and improve optimality in a multi-stage robust decision-making framework. In addition, since existing hourly resolution scheduling is unable to track fast and frequent variations in RES, the proposed framework is multi-resolution including both hourly and sub-hourly resolutions. Moreover, unlike conventional deterministic flexible ramp reserve procurement, a robust variability-oriented reserve scheduling is presented to determine adequate while economic and technically deliverable flexible ramp reserves from both generation and demand sides. The effectiveness of the proposed model is illustrated on the IEEE 24-bus reliability test system.

The WASCAL Hydrometeorological Observatory in the Sudan Savanna of Burkina Faso and Ghana

Core Ideas A new hydrometeorological observatory was established for the Sudan Savanna. More than 30 hydrometeorological variables in subhourly resolution are provided. Water, energy, and CO2 fluxes are monitored along a land use change gradient. The data form the basis for an improved impact assessment of environmental changes. It is an excellent platform for future field and modeling studies in West Africa. Watersheds with rich hydrometeorological equipment are still very limited in West Africa but are essential for an improved analysis of environmental changes and their impacts in this region. This study gives an overview of a novel hydrometeorological observatory that was established for two mesoscale watersheds in the Sudan Savanna of Southern Burkina Faso and Northern Ghana as part of the West African Science Service Centre on Climate Change and Adapted Land Use (WASCAL) program. The study area is characterized by severe land cover changes due to a strongly increasing demand of agricultural land. The observatory is designed for long‐term measurements of >30 hydrometeorological variables in subhourly resolution and further variables such as CO2. This information is complemented by long‐term daily measurements from national meteorological and hydrological networks, among several other datasets recently established for this region. A unique component of the observatory is a micrometeorological field experiment using eddy covariance stations implemented at three contrasting sites (near‐natural, cropland, and degraded grassland) to assess the impact of land cover changes on water, energy, and CO2 fluxes. The datasets of the observatory are needed by many modeling and field studies conducted in this region and are made available via the WASCAL database. Moreover, the observatory forms an excellent platform for future investigations and can be used as observational foundation for environmental observatories for an improved assessment of environmental changes and their socioeconomic impacts for the savanna regions of West Africa.

Development of a PV performance model for power output simulation at minutely resolution

Because of the transient and non-linear characteristics of solar irradiance, simulation at sub-hourly resolution, e.g., at the order of 10 min or less, becomes more essential. In this paper, a PV performance model at sub-hourly resolution based on global horizontal irradiance is developed and validated by the minutely on-site measurements under different weather conditions. Then, a simulation at minutely resolution is carried out for the year 2015 in Zhuhai, China. The results are compared with the simulation results at hourly resolution. It is found that simulation at minutely resolution can reflect the real characteristics of power output fluctuations. The differences of daily energy output between hourly and minutely resolution are in the range of −2%∼10%, and it seems that the higher values of daily clearness index will lead to higher values of daily differences. The differences of monthly and yearly energy output are −1∼7.5% and 2.46%, respectively. The differences for different time scales indicate that for a longer time period (e.g., one year), the simulated energy output at hourly resolution will go closer with that at minutely resolution.

Doubly stochastic Poisson pulse model for fine-scale rainfall

Stochastic rainfall models are widely used in hydrological studies because they provide a framework not only for deriving information about the characteristics of rainfall but also for generating precipitation inputs to simulation models whenever data are not available. A stochastic point process model based on a class of doubly stochastic Poisson processes is proposed to analyse fine-scale point rainfall time series. In this model, rain cells arrive according to a doubly stochastic Poisson process whose arrival rate is determined by a finite-state Markov chain. Each rain cell has a random lifetime. During the lifetime of each rain cell, instantaneous random depths of rainfall bursts (pulses) occur according to a Poisson process. The covariance structure of the point process of pulse occurrences is studied. Moment properties of the time series of accumulated rainfall in discrete time intervals are derived to model 5-min rainfall data, over a period of 69 years, from Germany. Second-moment as well as third-moment properties of the rainfall are considered. The results show that the proposed model is capable of reproducing rainfall properties well at various sub-hourly resolutions. Incorporation of third-order moment properties in estimation showed a clear improvement in fitting. A good fit to the extremes is found at larger resolutions, both at 12-h and 24-h levels, despite underestimation at 5-min aggregation. The proportion of dry intervals is studied by comparing the proportion of time intervals, from the observed and simulated data, with rainfall depth below small thresholds. A good agreement was found at 5-min aggregation and for larger aggregation levels a closer fit was obtained when the threshold was increased. A simulation study is presented to assess the performance of the estimation method.

Environmental and ecological controls of coral community metabolism on Palmyra Atoll

Accurate predictions of how coral reefs may respond to global climate change hinge on understanding the natural variability to which these ecosystems are exposed and to which they contribute. We present high-resolution estimates of net community calcification (NCC) and net community production (NCP) from Palmyra Atoll, an uninhabited, near-pristine coral reef ecosystem in the central Pacific. In August–October 2012, we employed a combination of Lagrangian and Eulerian frameworks to establish high spatial (~2.5 km2) and temporal (hourly) resolution coral community metabolic estimates. Lagrangian drifts, all conducted during daylight hours, resulted in NCC estimates of −51 to 116 mmol C m−2 h−1, although most NCC estimates were in the range of 0–40 mmol C m−2 h−1. Lagrangian drift NCP estimates ranged from −7 to 67 mmol C m−2 h−1. In the Eulerian setup, we present carbonate system parameters (dissolved inorganic carbon, total alkalinity, pH, and pCO2) at sub-hourly resolution through several day–night cycles and provide hourly NCC and NCP rate estimates. We compared diel cycles of all four carbonate system parameters to the offshore surface water (0–50 m depth) and show large departures from offshore surface water chemistry. Hourly Eulerian estimates of NCC aggregated over the entire study ranged from 14 to 53 mmol C m−2 h−1, showed substantial variability during daylight hours, and exhibited a diel cycle with elevated NCC in the afternoons and depressed, but positive, NCC at night. The Eulerian NCP range was very high (−55 to 177 mmol C m−2 h−1) and exhibited strong variability during daylight hours. Principal components analysis revealed that NCC and NCP were most closely aligned with diel cycle forcing, whereas the NCC/NCP ratio was most closely aligned with reef community composition. Our analysis demonstrates that ecological community composition is the primary determinant of coral reef biogeochemistry on a near-pristine reef and that reef biogeochemistry is likely to be responsive to human behaviors that alter community composition.

Reversing hydrology: Estimation of sub-hourly rainfall time-series from streamflow

A novel solution to the estimation of catchment rainfall at a sub-hourly resolution from measured streamflow is introduced and evaluated for two basins with markedly different flow pathways and rainfall regimes. It combines a continuous-time transfer function model with regularised derivative estimates obtained using a recursive method with capacity for handling missing data. The method has general implications for off-line estimation of unknown inputs as well as robust estimation of derivatives. It is compared with an existing approach using a range of model metrics, including residuals analysis and visuals; and is shown to recover the salient features of the observed, sub-hourly rainfall, sufficient to produce a precise estimate of streamflow, indistinguishable from the output of the catchment model in response to the observed rainfall data. Results indicate potential for use of this method in environment-related applications for periods lacking sub-hourly rainfall observations.

Assessing future climatic changes of rainfall extremes at small spatio-temporal scales

Climate change is expected to influence the occurrence and magnitude of rainfall extremes and hence the flood risks in cities. Major impacts of an increased pluvial flood risk are expected to occur at hourly and sub-hourly resolutions. This makes convective storms the dominant rainfall type in relation to urban flooding. The present study focuses on high-resolution regional climate model (RCM) skill in simulating sub-daily rainfall extremes. Temporal and spatial characteristics of output from three different RCM simulations with 25 km resolution are compared to point rainfall extremes estimated from observed data. The applied RCM data sets represent two different models and two different types of forcing. Temporal changes in observed extreme point rainfall are partly reproduced by the RCM RACMO when forced by ERA40 re-analysis data. Two ECHAM forced simulations show similar increases in the occurrence of rainfall extremes of over a 150-year period, but significantly different changes in the magnitudes. The physical processes behind convective rainfall extremes generate a distinctive spatial inter-site correlation structure for extreme events. All analysed RCM rainfall extremes, however, show a clear deviation from this correlation structure for sub-daily rainfalls, partly because RCM output represents areal rainfall intensities and partly due to well-known inadequacies in the convective parameterization of RCMs. The results highlight the problem urban designers are facing when using RCM output. The paper takes the first step towards a methodology by which RCM performance and other downscaling methods can be assessed in relation to the simulation of short-duration rainfall extremes.

Acid episodes in the Allt a'Mharcaidh, Scotland: an investigation based on sub-hourly monitoring data and climatic patterns

Abstract. Stream waters in the Allt a'Mharcaidh catchment (Cairngorms, Scotland) have been monitored for flow, conductivity and pH at sub-hourly resolution; and for a range of chemical, biological and physical parameters, less intensively, since the mid-1980s. The Allt a'Mharcaidh stream is subject to acidic events (pH<5.5) triggered by both hydrology and sea-salt inputs from the atmosphere. This paper investigates the drivers of these acidic events using variables derived from sub-hourly monitored data. It also examines the influence of the North Atlantic Oscillation (NAO) on episode severity. Sub-hourly datasets are used to derive multiple regression models expressing stream H+ concentration as a function of the sea-salt conductivity and the peak instantaneous flow rate amongst other explanatory variables. The relationship between sea-salt conductivity and the NAO is significant but hidden due to issues such as time lags and the influence of atmospheric patterns other than the NAO.