A Cached-based approach to enrich Stream data with master data

Abstract

An enriching of stream data with disk-based master data is common in many applications. Real-time data warehousing is one of these applications where stream data is customers transactions producing by operational data source(s). This stream data needs to enrich by disk-based master data before loading this into the data warehouse. To implement this enrichment operation a join operator is required. Normally we call it semi-stream join as it is performed between stream data and disk data. The join operator typically works under limited main memory and this memory is generally not large enough to hold the whole disk-based master data. Therefore, the relation is loaded into memory in partitions. A well-known join algorithm called MESHJOIN (Mesh Join) has been presented in the literature to implement the semi-stream join operation. However, the algorithm suffers with some limitations. In particular, the algorithm can be improved based on the characteristics of stream data e.g. skew in stream data. In this paper we address this issue by presenting a novel algorithm called Cached-based Stream-Disk Join (CSDJ). The algorithm exploits skew characteristic in stream data more appropriately and over performs existing MESHJOIN. We present results for Zip-fian distributions of the type that appear in many applications. We evaluate our algorithm using synthetic, TPC-H and real datasets. Our experiments show that CSDJ performs significantly better than MESHJOIN.