Query Execution Optimization Based on Incremental Update in Database Distributed Middleware

Abstract

Big data is often generated incrementally in the real word. Existing incremental query optimization is mainly used in the streaming data environment. Due to the constraints of real-time streaming data applications, existing incremental execution mechanisms is difficult to directly apply to large business-oriented data in a distributed environment. This paper proposes a query execution optimization method based on incremental update in database distributed middleware. First, the proposed method defines the Reference-Graph according to tables and their foreign key relationships, based on which a data partition strategy is provided to reduce data transmission quantity during query operation. In addition, the proposed method proposes an incremental update query execution strategy and incremental intermediate result preservation mechanism in distributed environment for non-aggregate and aggregate query respectively. The combination of data partition and incremental updating strategy reduces the query execution cost and enhance the performance of complex query operation significantly. Finally, the experimental results conducted on the benchmark dataset test and verify the effectiveness of the proposed method.