Streaming versus batch processing of sensor data in a hazardous weather detection system

Abstract

We present several discrete-time Markov queuing models to compare the performance of batch versus streaming processing of sensor data in a weather detection and monitoring system architecture. The first model assumes independent arrivals. The remaining mod- els assume correlated arrivals and demonstrate how dif- ferent scan strategies across multiple elevations impact system performance. We also show how the models are useful in dimensioning the computational resources of the system given workload arrival characteristics. We apply the models to several hypothetical scenarios with varying weather feature arrival and processing rates. We also eval- uate our models using processing runtime data obtained by running two NEXRAD algorithms on weather data from six airports. Our results show that for this particular application and reasonable system utilizations, delaying processing start time to perform batch processing does not adversely affect system performance or require significant, additional computational resources. More generally, our models show how system performance is dependent on the scan strategy and the burstiness of the sensor data. I. INTRODUCTION In this paper, we use discrete-time Markov queuing models to compare the performance of batch versus streaming processing of sensor data in a weather detec- tion and monitoring system. In particular, we model and analyze the feature detection subsystem of the Meteoro- logical Command and Control (MC&C) component of the proposed architecture for the Collaborative Adaptive Sensing of the Atmosphere (CASA) weather prediction system (1), (2). In the proposed MC&C, weather sen- sor data are input asynchronously to feature detection algorithms for a fixed time window during which the algorithms must process the data and deposit any de- tected features into a repository. The system then uses these features to redirect the sensors, and the process then repeats. Any features not deposited in the repository before the end of the current window are not used in the