Using Machine Learning with Pyspark for Solving a Big Data Problem: Searching for Particles in High Energy Physics with Big Mass (HEPMASS)

Abstract

AbstractThe search for exotic particles is one of the most challenging topics for physicists. This work aims to resolve the Big Data problem in the exotic particle area using the Apache Spark environment with the MLlib library. For this purpose, we analyze and discuss the performance of the following methods: Logistic Regression (LR), Decision Tree (DT), Random Forest (RF), and Gradient Boosted Tree (GBT). The results are very encouraging and prove that these proposed methods run efficiently with an Accuracy exceeding (83%). While the Gradient Boosted Tree (GBT) reaches a high AUC score 85%, better than Decision Tree (DT) and Random Forest. In this manuscript, we attempt to solve a big data problem using the “HEPMASS" dataset. Our data includes 10.5 million examples collected from the UCI website for the experimental phase. KeywordsExotic particlesPysparkMachine learning (ML)Logistic regression (LR)Decision tree (DT)Random forest (RF)Gradient boosted tree (GBT)Area under the curve (AUC)AccuracyComputation time (CT)