Load Shedding in MavStream: Analysis, Implementation, and Evaluation

Abstract

In data stream management systems (DSMSs), Quality of Service (or QoS) requirements, as specified by users, are extremely important. To satisfy QoS requirements throughout the life of a data stream, result characteristics need to be monitored at runtime and adjustments made continuously. It has been shown that in a DSMS, switching scheduling strategies at runtime can change tuple latency requirements. DSMSs also experience significant fluctuations in input rates (termed bursty inputs). In order to meet the QoS requirements in the presence of bursty inputs, a load shedding strategy is critical. This also entails monitoring of QoS measures at run-time to meet expected QoS requirements. This paper addresses load shedding issues for MavStream, a DSMS being developed at UT Arlington. To cope with situations where the arrival rates of input streams exceed the processing capacity of the system, we have incorporated load shedders into the query processing model. The runtime optimizer continually monitors the output and decides when to turn on the shedders and how much to shed. Choice of shedders is done to minimize the error in the output. Shedders have been incorporated as part of the buffers to minimize the overhead for load shedding. Finally, load shedders are activated and deactivated dynamically by the runtime optimizer. Both random and semantic load shedding techniques are supported to match application semantics.