A system based on Hadoop for radar data analysis

Abstract

Nowadays, radar technique is widely used in many applications, such as electronic warfare, weather prediction, navigation, and self-driving car. Large amounts of radar data has been generated by the wide use of radar technique. Analyzing radar data has a quite important role in daily life, as well as in military. Finding the frequent sequences in radar data is significant for radar data analysis. However, traditional analysis systems using standalones cannot process big data due to the four features of big data, namely, volume, velocity, variety and value. Many distributed frameworks are promising for processing large scale data sets, such as Hadoop and Spark. Therefore, to deal with the problem of finding frequent sequences from large amounts of radar data, we built a system based on Hadoop and Spark. With the combination of Hadoop and Spark, we can store big data, as well as analyze big radar data more easily. In the proposed system, Hadoop distributed file system offers stable data storage, and Spark offers efficient in-memory calculation. In this paper, a three-node Hadoop–Spark cluster was built to perform the distributed data mining algorithm. Additionally, to make the analysis of radar data accurate, we proposed ideas of preprocessing radar data and post processing mining results. Experimental results show that the system we proposed can analyze the large amounts of radar data efficiently and accurately.