A framework for Internet data real-time processing: A machine-learning approach

Abstract

Nowadays, the Internet Service Providers have to keep track of and in some cases to analyze for legal issues, a great amount of Internet data. Real-time big data processing and analysis introduce new challenges that must be addressed by system engineers. This is because: 1) traditional technologies exploiting databases are not designed to process a huge amount of data in real-time 2) classic machine learning algorithms implemented by widely adopted tools as Weka or R are not designed to perform “on the fly” analysis on streamed data. In this paper the authors propose an architecture that makes the real-time big data processing and analysis possible. The proposed architecture is based on two main components: a stream processing engine called Apache Storm and a framework called Yahoo SAMOA allowing to perform data analysis through distributed streaming machine learning algorithms. Our architecture is tested for Skype traffic recognition within network traffic generated by several Personal Computers in a streamed way. Experimental results have shown the effectiveness of proposed solution.