A scalable query materialization algorithm for interactive data exploration

Abstract

Data exploration is the process of competently digging out insights from stored data even if the user doesn't know or uncertain on what exactly he want. Iterative interactions of the user with search systems can help to achieve these goals; Interactive data exploration (IDE) is one such system, supporting data exploration by simply incorporating user feedback on retrieve data. These systems are the key ingredient of many discoveries and recall-oriented real-life applications such as Scientific Computing, Financial analysis, Social Analytics, Evidence-based medicine etc. IDE supports user's navigation through the query to query transition in data space, via exploratory session'. The exploratory session consists of often long, complex and analytical Queries. When processed against a large and multi-faceted data, hence consume a lot of time for processing. The user's original query can be decomposed into multiple candidate queries, ‘checkpoint queries’, and will be selected for materialization. In his paper, we introduced the notion of ‘checkpoint queries’, also discussed how heuristics query frequency and query result overlap ratio (QROR) are used in a greedy selection of these candidate queries from an exploratory session. It is observed, checkpoint queries proved a good decision for significant improvement in query processing time, as materialized checkpoint queries can be used in the query answering in query navigation. In this process, we face some daunting tasks; one key challenge is the selection of checkpoint queries for the materialization and how to improve the query reuse for query answering. In spite of the simplicity, our algorithm selects queries which give us better performance than views that selected by existing algorithms.