Spark implementation of the enhanced Scatter Search metaheuristic: Methodology and assessment

Abstract

Abstract Optimization problems arise nowadays in all disciplines, not only in the scientific area but also in the field of engineering or economics, and in many others. Currently, challenging optimization problems require solution methods that consume a significant amount of computational resources. The application of High-Performance Computing techniques is a common approach to obtain efficient implementations in traditional parallel computing systems. However, more recent approaches are exploring distributed programming frameworks developed in recent years to achieve efficient computations on clusters and cloud systems. In this paper we present a parallel implementation of the enhanced Scatter Search metaheuristic using Spark. The parallel program was obtained as a particularization of a general software framework we developed to support different realisations of the Scatter Search metaheuristic. The aim of this paper is to provide helpful guidance to readers interested in applying, or developing their own, parallel metaheuristics to solve challenging problems in the Cloud. With the twofold objective of demonstrating the potential of the parallelization with Spark and also of studying the factors that influence the performance of the solution, the proposal has been thoroughly evaluated on two different platforms, a cluster and a cloud platform, using a representative set of parameter estimation problems in the field of Computational Systems Biology.