Parallel Random Forest Algorithm Optimization Based on Maximal Information Coefficient

Abstract

In order to solve the problem that the traditional random forest algorithm runs too long or cannot be executed facing massive data, meanwhile in order to solve the problem that some redundant features are added to the training process and some strong expressive features are not selected when the traditional random forest algorithm randomly chooses features. A random forest algorithm based on maximum information coefficient (MIC)is proposed, and the algorithm is parallelized on the Spark platform. Firstly, MIC is used to rank each feature and the features are divided into three interval: high correlation interval, middle correlation interval and low correlation interval. In the process of constructing a single decision tree, the features of low correlation interval are deleted. Then, all the features of high correlation interval and the randomly selected features of middle correlation interval are selected to form a new feature subset to build the decision tree. Finally, the parallelization of the algorithm is implemented based on Spark. The experimental results show that the proposed algorithm has a certain improvement in accuracy and stability compared with the traditional random forest algorithm.