Chapter 45 - Effective Change Detection Using Sampling

Abstract

This chapter proposes three sampling-based download policies that can identify more changed data items effectively. For a large-scale data-intensive environment, such as the World Wide Web or data warehousing, one often makes local copies of remote data sources. Due to limited network and computational resources, however, it is often difficult to monitor the sources constantly to check for changes and to download changed data items to the copies. Many applications often make local copies of remote data sources. For instance, a data warehouse may copy remote sales and transaction records for local analysis. Similarly, a Web search engine copies a subset of the Web and indexes them to help users access Web pages. In many cases, the remote sources are updated independently of the local copies, so one must periodically poll and download data from the sources to detect changes and incorporate them to the copies. Change detection and download is often performed in batch at regular intervals, typically during off-peak hours, to avoid interference with the main tasks that the sources and/or clients perform. As the size of the data grows, however, detecting changes and incorporating them to the copies become increasingly difficult. Due to limited network and computational resources, one may not be able to check every data item in the data sources within the limited time window, so one may miss certain changes at the sources.