Disaggregation Algorithm Research Articles

A joint modelling framework for daily extremes of river discharge and precipitation in urban areas

AbstractHuman settlements are often at risk from multiple hydro‐meteorological hazards, which include fluvial floods, short‐time extreme precipitation (leading to ‘pluvial’ floods) or coastal floods. In the past, considerable scientific effort has been devoted to assessing fluvial floods. Only recently have methods been developed to assess the hazard and risk originating from pluvial phenomena, whereas little effort has been dedicated to joint approaches. The aim of this study was to develop a joint modelling framework for simulating daily extremes of river discharge and precipitation in urban areas. The basic framework is based on daily observations coupled with a novel precipitation disaggregation algorithm using nearest neighbour resampling combined with the method of fragments to overcome data limitations and facilitate its transferability. The framework generates dependent time series of river discharge and urban precipitation that allow for the identification of fluvial flood days (daily peak discharge), days of extreme precipitation potentially leading to pluvial phenomena (maximum hourly precipitation) and combined fluvial–pluvial flood days (combined time series). Critical thresholds for hourly extreme precipitation were derived from insurance and fire service data.

A Dynamic Disaggregation Approach to Approximate Linear Programs for Network Revenue Management

The linear programming approach to approximate dynamic programming has received considerable attention in the recent network revenue management (RM) literature. A major challenge of the approach lies in solving the resulting approximate linear programs (ALPs), which often have a huge number of constraints and/or variables. Starting from a recently developed compact affine ALP for network RM, we develop a novel dynamic disaggregation algorithm to solve the problem, which combines column and constraint generation and exploits the structure of the underlying problem. We show that the formulation can be further tightened by considering structural properties satisfied by an optimal solution. We prove that the sum of dynamic bid‐prices across resources is concave over time. We also give a counterexample to demonstrate that the dynamic bid‐prices of individual resources are not concave in general. Numerical experiments demonstrate that dynamic disaggregation is often orders of magnitude faster than existing algorithms in the literature for problem instances with and without choice. In addition, adding the concavity constraints can further speed up the algorithm, often by an order of magnitude, for problem instances with choice.

Weekly Streamflow Forecasting Using a Statistical Disaggregation Model for the Upper Blue Nile Basin, Ethiopia

AbstractAccurately forecasting the weekly seasonal streamflow of the Upper Blue Nile basin (UBNB) in Ethiopia is essential for managing large-scale water projects of Nile basinwide countries. A wavelet-based, artificial neural network calibrated by genetic algorithm (ANN–GA) model and a statistical disaggregation algorithm were integrated to forecast weekly streamflow of the UBNB. The July to October (JASO) streamflow of the El Diem station of UBNB shows strong interannual oscillations prior to the 1920s and after 1990s. Two ANN-GA models were developed to forecast the UBNB JASO streamflow, the first one using the February to May (FMAM) seasonal sea surface temperature (SST) of the global oceans as predictors to directly forecast JASO streamflow, while the second, a hybrid model, is developed to forecast JASO streamflow from two sets of predictors, which consist of FMAM SST and the July to September (JJAS) seasonal rainfall previously forecasted by the wavelet-based, ANN-GA also driven by FMAM SST as pred...

What Can Acreage Allocations Say about Crop Supply Elasticities? A Convex Programming Approach to Supply Response Disaggregation

AbstractThis article has two goals. The first is to extend previous results regarding calibration of land‐constrained programming models of agricultural supply against supply elasticities to the general case of multiple constraints. The second goal is to demonstrate how the resulting calibration conditions can be used as a source of identification to disaggregate crop supply elasticities down to the level at which static information on acreage allocations is available. We propose an information‐based disaggregation algorithm to systematically generate regionalised elasticities from a single prior, and provide an empirical illustration.

A Novel Approach Based on Finite-State Machines with Fuzzy Transitions for Nonintrusive Home Appliance Monitoring

Recent studies have highlighted that a significant part of the electrical energy consumption in residential buildings is caused by an improper use of home appliances. The development of low-cost systems for profiling the consumption of electric appliances can play a key role in stimulating the users to adopt adequate policies for energy saving. In this paper, we describe a novel methodology for extracting the power consumption of each appliance deployed in a domestic environment from the aggregate measures collected by a single smart meter. In order to coarsely describe how each type of appliance works, we use finite-state machines (FSMs) based on fuzzy transitions. An ad-hoc disaggregation algorithm exploits a database of these FSMs for, at each meaningful variation in real and reactive aggregate powers, hypothesizing possible configurations of active appliances. This set of configurations is concurrently managed by the algorithm which, whenever requested, outputs the configuration with the highest confidence with respect to the sequence of detected events. We implemented a prototype of a monitoring system based on the proposed methodology and installed it in a real domestic scenario. We discuss an experiment in which 11 appliances were connected to the same circuit and the aggregate power consumption was measured by a smart meter for approximately 12 h. At the end of the experiment, only two possible configurations were outputs from the system, including the correct one.

Real‐time load disaggregation algorithm using particle‐based distribution truncation with state occupancy model

A new particle-based distribution truncation method with a duration-dependent hidden semi-Markov model for non-intrusive appliance load monitoring is presented. Unlike earlier works, the approach keeps track of a set of states without prematurely pruning away intermediately ranked states. It also enables appliance state duration characteristics to be incorporated in a straightforward manner. Results show that the approach outperforms both the Viterbi algorithm and conventional particle-filtering methods.

An analysis of the optimization disaggregation algorithm in the estimation related to energy consumption of appliances in buildings

Generally speaking, energy monitoring system plays an important role in both energy use benchmarking and energy conservation service for large-scale buildings. However, it is often necessary for us to disaggregate the metered data according to types of energy use. Aiming at eliminating unclear concepts between monitoring and disaggregation and incorporating existing algorithms’ advantages into the process in which this estimation is generated, the optimal disaggregation algorithm is developed. Based on the assumption of mutually independent ends, a Bayes process is degenerated into maximization likelihood estimation. Following this, the process is expressed by the least squares. With a new algorithm, an unbiased and overall convergent posterior estimation is provided for each terminal category, which is powerfully supported by theoretical analyses and numerical calculation. Actually, it is proved that the disaggregation algorithm is quite appropriate and cost-efficient for sub-item energy monitoring systems in large-scale buildings. Meanwhile, this thesis also focuses on discussing the conditional probability of overall lessening estimation errors specifically. In this way, posterior estimations of overall accuracy are available when prior estimations are concordantly biased with much different accuracy.

Soil moisture at watershed scale: Remote sensing techniques

Summary Soil moisture at high spatial resolution is required for various land processes related studies. However, currently the resolution of passive microwave retrieved soil moisture is low – around 25 km. To solve this problem, a soil moisture disaggregation algorithm based on thermal inertia relationship between daily temperature change and average soil moisture modulated by vegetation conditions has been formulated. This algorithm was applied to the AMSR-E (Advanced Microwave Scanning Radiometer – Earth Observing System) as well as SMOS (Soil Moisture and Ocean Salinity satellite) to produce the 1 km downscaled soil moisture over the Little Washita Watershed in Oklahoma for the growing season in 2010 and 2011.The disaggregated soil moisture has been compared to in situ observations. The results of this approach are very encouraging.

Performance evaluation of a water resources system under varying climatic conditions: Reliability, Resilience, Vulnerability and beyond

Summary As introduced by Hashimoto et al. (1982), Reliability, Resilience, and Vulnerability (RRV) metrics measure different aspects of a water resources system performance. Together, RRV metrics provide one of the most comprehensive approaches for analyzing the probability of success or failure of a system, the rate of recovery (or rebound) of a system from unsatisfactory states, as well as quantifying the expected consequence of being in unsatisfactory states for extended periods. Assessing these comprehensive metrics at current (baseline) and future scenarios provide insight into system performance in changing or varying climatic conditions. Such an approach makes it possible to analyze different scenarios that could include specific mitigation or adaptation strategies to accommodate a varying climate. The method requires a subjective decision defining what constitutes an “unsatisfactory state” depending on acceptable risks. The application of this methodology is demonstrated using Tampa Bay Water’s Enhanced Surface Water System. In this case, for each scenario, a thousand ensembles of 300-years of monthly stream flow traces were first generated by a multi-site rainfall/runoff model. Second, a novel nonlinear disaggregation algorithm was developed to translate monthly outputs into daily values. The daily stream flow traces and their derivatives are then used to drive complex operational models that produce several system variables (e.g., permitted river withdrawals, reservoir storage volumes, and treatment plant production rates) at different locations. Outputs from the operational model were then used to define criteria over which the RRV and other metrics were evaluated. Several mitigation scenarios such as treatment and reservoir capacity expansion, as well as adaptation through operational changes were considered to evaluate system performance under varying climatic conditions. The approach highlights the benefits of comprehensive system performance metrics that are easy to understand by decision makers and stake holders and demonstrates the implementation of seemingly intractable ensemble size and simulation length in a distributed computing environment.

Self-calibrated evaporation-based disaggregation of SMOS soil moisture: An evaluation study at 3 km and 100 m resolution in Catalunya, Spain

A disaggregation algorithm is applied to 40km resolution SMOS (Soil Moisture and Ocean Salinity) surface soil moisture using 1km resolution MODIS (MODerature resolution Imaging Spectroradiometer), 90m resolution ASTER (Advanced Spaceborne Thermal Emission and Reflection radiometer), and 60m resolution Landsat-7 data. DISPATCH (DISaggregation based on Physical And Theoretical scale CHange) distributes high-resolution soil moisture around the low-resolution observed mean value using the instantaneous spatial link between optical-derived soil evaporative efficiency (ratio of actual to potential evaporation) and near-surface soil moisture. The objective is three-fold: (i) evaluating DISPATCH at a range of spatial resolutions using readily available multi-sensor thermal data, (ii) deriving a robust calibration procedure solely based on remotely sensed data, and (iii) testing the linear or nonlinear behavior of soil evaporative efficiency. Disaggregated soil moisture is compared with the 0–5cm in situ measurements collected each month from April to October 2011 in a 20km square spanning an irrigated and dry land area in Catalunya, Spain. The target downscaling resolution is set to 3km using MODIS data and to 100m using ASTER and Landsat data. When comparing 40km SMOS, 3km disaggregated and 100m disaggregated data with the in situ measurements aggregated at corresponding resolution, results indicate that DISPATCH improves the spatio-temporal correlation with in situ measurements at both 3km and 100m resolutions. A yearly calibration of DISPATCH is more efficient than a daily calibration. Assuming a linear soil evaporative efficiency model is adequate at kilometric resolution. At 100m resolution, the very high spatial variability in the irrigated area makes the linear approximation poorer. By accounting for non-linearity effects, the slope of the linear regression between disaggregated and in situ measurements is increased from 0.2 to 0.5. Such a multi-sensor remote sensing approach has potential for operational multi-resolution monitoring of surface soil moisture and is likely to help parameterize soil evaporation at integrated spatial scales.

Non-Intrusive Load Monitoring Approaches for Disaggregated Energy Sensing: A Survey

Appliance Load Monitoring (ALM) is essential for energy management solutions, allowing them to obtain appliance-specific energy consumption statistics that can further be used to devise load scheduling strategies for optimal energy utilization. Fine-grained energy monitoring can be achieved by deploying smart power outlets on every device of interest; however it incurs extra hardware cost and installation complexity. Non-Intrusive Load Monitoring (NILM) is an attractive method for energy disaggregation, as it can discern devices from the aggregated data acquired from a single point of measurement. This paper provides a comprehensive overview of NILM system and its associated methods and techniques used for disaggregated energy sensing. We review the state-of-the art load signatures and disaggregation algorithms used for appliance recognition and highlight challenges and future research directions.

A conditional disaggregation algorithm for generating fine time-scale rainfall data in a warmer climate

Summary This paper describes an algorithm for disaggregating daily rainfall into sub-daily rainfall ‘fragments’ (fine-resolution rainfall sequences) under a future, warmer climate. The algorithm uses a combined generalised additive model (GAM) and method of fragments (MoFs) framework to resample sub-daily rainfall fragments from the historical record conditional on daily rainfall amount and a range of atmospheric covariates. The rationale is that as the atmosphere warms, future rainfall patterns will be more reflective of historical rainfall patterns corresponding to warmer days at the same location, or to locations which have an atmospheric profile more reflective of expected future climate. It was found that the daily to sub-daily scaling relationship varied significantly by season and by location, with rainfall patterns on warmer seasons or at warmer locations typically showing more intense rainfall occurring over shorter periods compared with cooler seasons and stations. Importantly, by regressing against atmospheric covariates such as temperature, this effect was substantially reduced, suggesting that the approach may also be valid when extrapolating to a future climate. The GAM–MoF algorithm was then applied to nine stations around Australia, with the results showing that relative to the daily rainfall amount, the maximum intensity of short duration rainfall increased by between 4.1% and 13.4% per degree change in temperature for the maximum six minute burst, and by between 3.1% and 6.8% for the maximum 1 h burst. The fraction of each wet day with no rainfall also increased by between 1.5% and 3.5%. This highlights that a significant proportion of the change to the distribution of rainfall is likely to occur at sub-daily timescales, with important implications for many hydrological systems.

Is disaggregation the holy grail of energy efficiency? The case of electricity

This paper aims to address two timely energy problems. First, significant low-cost energy reductions can be made in the residential and commercial sectors, but these savings have not been achievable to date. Second, billions of dollars are being spent to install smart meters, yet the energy saving and financial benefits of this infrastructure – without careful consideration of the human element – will not reach its full potential. We believe that we can address these problems by strategically marrying them, using disaggregation. Disaggregation refers to a set of statistical approaches for extracting end-use and/or appliance level data from an aggregate, or whole-building, energy signal. In this paper, we explain how appliance level data affords numerous benefits, and why using the algorithms in conjunction with smart meters is the most cost-effective and scalable solution for getting this data. We review disaggregation algorithms and their requirements, and evaluate the extent to which smart meters can meet those requirements. Research, technology, and policy recommendations are also outlined.

Simple method for retrieving leaf area index from Landsat using MODIS leaf area index products as reference

Leaf area index (LAI) is a key parameter in most land surface models. Models that operate at multiple spatial scales may require consistent LAI inputs at different spatial resolutions or from different sensors. For example, the atmosphere-land exchange inverse model and associated disaggregation algorithm (DisALEXI) use the moderate resolution imaging spectroradiometer (MODIS) LAI product to model fluxes at regional scales (1- to 10-km grid resolution), and Landsat-based LAI to disaggregate to field scale (30-m grid). In order to make a MODIS-consistent LAI product from Landsat imagery for this combined scheme, a simple reference-based regression tree approach was developed. This approach uses homogeneous and high-quality LAI retrievals from MODIS as references to develop a regression tree relating these MODIS LAI samples to Landsat surface reflectances. Results show that the approach can produce accurate estimates of LAI from Landsat, as evaluated using field measurements collected during the soil moisture experiment of 2002, conducted in central Iowa during a period of rapid vegetation growth. The coefficient of determination (r2) computed between Landsat retrievals and field measurements was 0.94 at the field scale, with an overall mean bias error (MBE) of −0.07 and mean absolute difference (MAD) of 0.23. MAD values of 0.17 and 0.32 were obtained for low to moderate LAI (0-3) and high LAI (>3), respectively, with some underestimation for the high LAI (MBE = −0.28). The LAI maps retrieved from Landsat were consistent with the MODIS estimates when aggregated to coarser scales. MAD computed between Landsat- and MODIS-derived LAI ranged from 0.07 to 0.83 for different Landsat dates, with no significant bias compared to MODIS high-quality retrievals. This approach demonstrates a simple framework for producing MODIS-consistent LAI from Landsat data for modeling the land surface at different spatial scales.

Improving device-level electricity consumption breakdowns in private households using ON/OFF events

Smart meters can measure the electricity consumption of a household at a fine temporal granularity. By adequately processing this aggregated data an estimation of the consumption of individual appliances can be retrieved and used to provide novel services, such as personalized recommendations on how to reduce the overall energy consumption of the household. In this paper, we build upon existing work in consumption data disaggregation and consider smart meter data along with additional information made available by networked sensors and household appliances. In particular, we investigate the use of ON/OFF events, which signal when appliances have been turned on or off, to improve the accuracy of a state-of-the art disaggregation algorithm that uses such events along with smart meter data to estimate the consumption of single appliances. Our results, obtained by applying the algorithm to a publicly available dataset, show that the accuracy of the algorithm quickly deteriorates as the number of available ON/OFF events decreases. We thus suggest possible countermeasures to cope with this limitation and to provide accurate electricity consumption breakdowns in private households.

Test System for Disaggregation Algorithms for use in Smart Meters

In this paper a test system for verifying disaggregation algorithms is presented. The main part of the test system is a test set generator. Based on the model of a real voltage source, the generator generates test sets. With these test sets the quality of the developed algorithms can be analyzed. These algorithms will be developed for the implementation in a smart meter. The smart meter system is used for the disaggregation of electrical appliances in residential buildings. This allows to be closed on energy consumption of the individual appliance. The development of a complete smart meter system is the goal of the research project “Smart Metering” at the University of Furtwangen.

IHMS—Integrated Hydrological Modelling System. Part 1. Hydrological processes and general structure

AbstractA newly Integrated Hydrological Modelling System (IHMS) has been developed to study the impact of changes in climate, land use and water management on groundwater and seawater intrusion (SWI) into coastal areas. The system represents the combination of three models, which can, if required, be run separately. It has been designed to assess the combined impact of climate, land use and groundwater abstraction changes on river, drainage and groundwater flows, groundwater levels and, where appropriate, SWI. The approach is interdisciplinary and reflects an integrated water management approach. The system comprises three packages: the Distributed Catchment Scale Model (DiCaSM), MODFLOW (96 and 2000) and SWI models. In addition to estimating all water balance components, DiCaSM, produces the recharge data that are used as input to the groundwater flow model of the US Geological Survey, MODFLOW. The latter subsequently generates the head distribution and groundwater flows that are used as input to the SWI model, SWI. Thus, any changes in land use, rainfall, water management, abstraction, etc. at the surface are first handled by DiCaSM, then by MODFLOW and finally by the SWI. The three models operate at different spatial and temporal scales and a facility (interface utilities between models) to aggregate/disaggregate input/output data to meet a desired spatial and temporal scale was developed allowing smooth and easy communication between the three models. As MODFLOW and SWI are published and in the public domain, this article focuses on DiCaSM, the newly developed unsaturated zone DiCaSM and equally important the interfacing utilities between the three models. DiCaSM simulates a number of hydrological processes: rainfall interception, evapotranspiration, surface runoff, infiltration, soil water movement in the root zone, plant water uptake, crop growth, stream flow and groundwater recharge. Input requirements include distributed data sets of rainfall, land use, soil types and digital terrain; climate data input can be either distributed or non‐distributed. The model produces distributed and time series output of all water balance components including potential evapotranspiration, actual evapotranspiration, rainfall interception, infiltration, plant water uptake, transpiration, soil water content, soil moisture (SM) deficit, groundwater recharge rate, stream flow and surface runoff. This article focuses on details of the hydrological processes and the various equations used in DiCaSM, as well as the nature of the interface to the MODFLOW and SWI models. Furthermore, the results of preliminary tests of DiCaSM are reported; these include tests related to the ability of the model to predict the SM content of surface and subsurface soil layers, as well as groundwater levels. The latter demonstrates how the groundwater recharge calculated from DiCaSM can be used as input into the groundwater model MODFLOW using aggregation and disaggregation algorithms (built into the interface utility). SWI has also been run successfully with hypothetical examples and was able to reproduce the results of some of the original examples of Bakker and Schaars (2005). In the subsequent articles, the results of applications to different catchments will be reported. Copyright © 2010 John Wiley & Sons, Ltd.

An enhanced nonparametric streamflow disaggregation model with genetic algorithm

Stochastic streamflow generation is generally utilized for planning and management of water resources systems. For this purpose, a number of parametric and nonparametric models have been suggested in literature. Among them, temporal and spatial disaggregation approaches play an important role particularly to make sure that historical variance‐covariance properties are preserved at various temporal and spatial scales. In this paper, we review the underlying features of existing nonparametric disaggregation methods, identify some of their pros and cons, and propose a disaggregation algorithm that is capable of surmounting some of the shortcomings of the current models. The proposed models hinge on k‐nearest neighbor resampling, the accurate adjusting procedure, and a genetic algorithm. The models have been tested and compared to an existing nonparametric disaggregation approach using data of the Colorado River system. It has been shown that the model is capable of (1) reproducing the season‐to‐season correlations including the correlation between the last season of the previous year and the first season of the current year, (2) minimizing or avoiding the generation of flow patterns across the year that are literally the same as those of the historical records, and (3) minimizing or avoiding the generation of negative flows. In addition, it is applicable to intermittent river regimes.

An improved algorithm for disaggregating microwave-derived soil moisture based on red, near-infrared and thermal-infrared data

Accurate high-resolution soil moisture data are needed for a range of agricultural and hydrologic activities. To improve the spatial resolution of ∼40km resolution passive microwave-derived soil moisture, a methodology based on 1km resolution MODIS (MODerate resolution Imaging Spectroradiometer) red, near-infrared and thermal-infrared data has been implemented at 4km resolution. The three components of that method are (i) fractional vegetation cover, (ii) soil evaporative efficiency (defined as the ratio of actual to potential evaporation) and (iii) a downscaling relationship. In this paper, 36 different disaggregation algorithms are built from 3 fractional vegetation cover formulations, 3 soil evaporative efficiency models, and 4 downscaling relationships. All algorithms differ with regard to the representation of the nonlinear relationship between microwave-derived soil moisture and optical-derived soil evaporative efficiency. Airborne L-band data collected over an Australian agricultural area are used to both generate ∼40km resolution microwave pixels and verify disaggregation results at 4km resolution. Among the 36 disaggregation algorithms, one is identified as being more robust (insensitive to soil, vegetation and atmospheric variables) than the others with a mean slope between MODIS-disaggregated and L-band derived soil moisture of 0.94. The robustness of that algorithm is notably assessed by comparing the disaggregation results obtained using composited (averaged) Terra and Aqua MODIS data, and using data from Terra and Aqua separately. The error on disaggregated soil moisture is systematically reduced by compositing daily Terra and Aqua data with an error of 0.012vol./vol.

Disaggregation Method for an Aggregate Traffic Flow Management Model

A linear time-varying aggregate traffic flowmodel can be used to develop traffic flowmanagement strategies using optimization algorithms. However, there are few methods available in the literature to translate these aggregate solutions into practical control actions involving individual aircraft. In this paper, a computationally efficient disaggregation algorithm is proposed by employing a series of linear program and mixed integer linear program methods, which converts an aggregate (flow-based) solution to a flight-specific control action. Numerical results generated by the optimization method and the disaggregation algorithm are presented and illustrated by applying them to generate traffic flow management schedules for a typical day in the U.S. National Airspace System.