A Study of Big Data Analytics using Apache Spark with Python and Scala

Abstract

Data is generated by humans every day via various sources such as Instagram, Facebook, Twitter, Google, etc at a rate of 2.5 quintillion bytes with high volume, high speed and high variety. When this huge volume of data with high velocity is handled by the traditional approaches, it becomes inefficient and time-consuming. Apache Spark technology has been used that is an open-source in-memory clusters computing system for fast processing. This paper introduces a brief study of Big Data Analytics and Apache Spark which consists of characteristics (7V’s) of big data, tools and application areas for big data analytics, as well as Apache Spark Ecosystem including components, libraries and cluster managers that is deployment modes of Apache spark. Furthermore, this also presents a comparative study of Python and Scala programming languages with many parameters in the aspect of Apache Spark. This comparative study will help to identify which programming language like Python and Scala are suitable for Apache Spark technology. As result, both Python and Scala programming languages are suitable for Apache Spark, however language choice for programming in Apache Spark is depending on the features that best suited the needs of the project since each one has its own advantages and disadvantages. The main purpose of this paper is to make things easier for the programmer in the selection of programming languages in Apache Spark-based on their project.