A fast and highly scalable frequent pattern mining algorithm

Abstract

As the use of big data and its potential benefits become more widespread, public and private organizations around the world have realized the imperative of incorporating comprehensive and robust technologies into their business processes. In particular, companies are implementing more and more intelligent systems into their business processes, resulting in an exponential increase in the amount of data being collected and used in everyday life on a quarterly basis. Therefore, an algorithm that can efficiently explore the frequent patterns from big data will be able to productively analyze and utilize the generated data to better optimize resources and even develop new business models. Thus, in this urban data era, it is not only essential to make more efficient and accurate decisions to improve business profitability and customer satisfaction, but also a highly challenging core issue. In this study, a new approach, scalable FP mining, is proposed for the first time to achieve a certain level of performance and more efficient memory utilization despite large amounts of data. Experiment results for different data characteristics show that the best performance of the proposed method requires only 48.9% of the DP algorithm's execution time. In further experiments with increased exploration difficulty, the optimal execution time is only 37%. On specific associated databases, the proposed method maintains stable and excellent performance with 12.7% for different data characteristics. The experimental results with increasing exploration difficulty at each stage demonstrate the stability, robustness, and reliability of the proposed SFP, especially under high data complexity and exploration difficulty.