Dolphin-political optimized tversky index based feature selection in spark architecture for clustering big data

Abstract

BackgroundThe analytics of big data has gained immense attention over the classical data-processing techniques which engaged in mining the hidden patterns from huge data known as big data. For relieving the computational complexity, the clustering technique is considered as an imperative part. MethodsThis paper presents an optimization-driven technique, namely Dolphin Political Optimizer (DPO) for clustering big data with the Spark model. The developed clustering model is devised using spark architecture that contains master and slave nodes for accomplishing the clustering tasks. The input big data are large-sized data and hence are required to partition the input data into different blocks with varying size. The selection of features is done using the proposed Tversky-based DPO, which is obtained by incorporating the Tversky index in DPO in the slave node. Here, the proposed Dolphin Political Optimizer (DPO) is devised by combining Dolphin Echolocation (DE) and Political Optimizer (PO) respectively. The data augmentation is done using oversampling. The clustering of big data is done using entropy weighting power k-Means clustering where in the weight is updated using proposed DPO algorithm. ResultThe assessment of the proposed DPO is done using clustering accuracy, Jaccard coefficient, rand coefficient, Silhouette coefficient.The proposed DPO outperformed with the highest clustering accuracy of 0.937, Jaccard coefficient of 0.670, Rand coefficient of 0.851, and the highest silhouette coefficient of 0.769. ConclusionThis approach demonstrated improved robustness and produced the world's best optimal solution. When comparing with existing methods the proposed Tversky-based DPO offered effective performance.