Testbeds, Attacks, and Dataset Generation for Big Data Cluster: A System Application for Big Data Platform Security Analysis

Abstract

A big data cluster consists of number of network-connected computers. A big data cluster offers a huge data store and processing power. End users submit both data and application to the cluster. All the computers called nodes in the cluster work together to give the result from the data. During data processing, lots of process run on different nodes and exchange data. The data exchange is done via regular network protocols. During processing, one or multiple computers may not participate well due to its bad hardware or operating system health. Some computers may receive known network attack like DOS and thus slow down the performance of the cluster. Some other computers may receive unknown attacks generated by the big data job itself. Therefore, the system requires a mechanism to detect such nodes under attack or the nodes generating attacks and isolate thereafter. To detect this attack, we need to analyze the cumulative network traffic of all the nodes in the cluster. Therefore, we must collect such network traffic of all the nodes participating in data processing job simultaneously. This work is to present an efficient testbed for external or internal attack generation and dataset creation for different attacks. The proposed architecture captures network traffic from all nodes of the cluster and stores them for attack detection in near future.