Probabilistic Programming and Big Data

Abstract

The advent of the Internet in the late 1990s led to the increased flood of data termed Big Data. To derive meaningful value from big data, specialized tools and techniques are required. These tools and techniques are categorized under data management and data analysis. Under data analysis, predictive tools and techniques exists. These predictive tools and techniques use statistical models and machine learning algorithms to predict future events. Models are developed using probability theories such as Bayesian networks. However, development of probabilistic models require extreme technical expertise and it is a difficult task to model complex real-life situations. Thus, the emergence of probabilistic programming. The idea of probabilistic programming is new and its potential in AI and big data processing is important. This paper presents discussions on probabilistic reasoning and probabilistic programming with respect to big data. An investigation of the potential of probabilistic programming in big data is also presented by conducting a search through literature to find available big data solutions that use probabilistic programming. This search found one solution called InferSpark built on top of Apache Spark to process big data. This is an indication that more big data applications that uses the concept of probabilistic programming needs to be done.