Prediction-based load shedding for burst data streams

Abstract

Overload management has become very important in telecommunication networks, especially in the case of monitoring network elements that generate time-varying and burst data streams. Efficient overload management improves the quality of provided services, reduces revenue leakage, and minimizes network failure detection time. We argue that in order to improve the quality of bursty data stream processing, data stream management systems (DSMS) should be introduced into network management infrastructures and should monitor and optimize the execution of data stream queries through load-shedding techniques. We introduce a new mechanism for data prioritization and implement a new load-shedding framework, partially based on the popular TelegraphCQ DSMS. Our approach significantly improves the accuracy of data stream query results by using a prediction module to rank the values using preferential ranks. The prediction module tracks and registers all symptoms of overload, and either prevents overload before it occurs, or minimizes its effects. We present empirical evidence showing that our solution significantly improves data stream query accuracy. The effectiveness of the framework has been verified using both artificial and real data streams collected from network elements.