Scalable formal concept analysis algorithms for large datasets using Spark

Abstract

In the process of knowledge discovery and representation in large datasets using formal concept analysis, complexity plays a major role in identifying the formal concepts and constructing the concept lattice (digraph of the concepts). For identifying the formal concepts and constructing the digraph from the identified concepts in large datasets, various distributed algorithms are available. However, the existing distributed algorithms are not well suited for concept generation, because the generation of concepts is an iterative process. Existing algorithms are implemented using distributed frameworks like MapReduce and Open MP. These frameworks are not appropriate for iterative applications. Hence, there is a need for efficient distributed algorithms for both formal concept generation and concept lattice digraph construction in large formal contexts. In this paper, we present efficient algorithms using Apache Spark. The various performance metrics used in evaluation prove that the proposed algorithms are more efficient for concept generation and lattice graph construction than existing algorithms.