View Maintenance Research Articles

View disassembly: A rewrite that extracts portions of views

We explore a new form of view rewrite called view disassembly. The objective is to rewrite views in order to “remove” certain sub-views (or unfoldings) of the view. This becomes pertinent for complex views which may be defined over other views and which may involve union. Such complex views arise necessarily in environments such as data warehousing and mediation over heterogeneous databases. View disassembly can be used for view and query optimization, preserving data security, making use of cached queries and materialized views, and view maintenance. We provide computational complexity results of view disassembly. One question is whether the unfoldings to be removed effectively cover the view, meaning that the disassembled view is effectively the empty view, evaluating to the empty table. We illustrate the complexity to determine when a collection of unfoldings cover the view definition. The problem is NP- hard with respect to the number of unfoldings to remove, but not with respect to the size (complexity) of the view definition. We next consider rewrites optimal in the size of the rewritten (disassembled) view. We prove that this task is NP- hard for a special class of views, but this time NP- hard in a worse way: it is NP- hard this time with respect to the size of the view definition in addition to the number of unfoldings to be removed. In general, we suspect the problem is computationally even harder, and we show that the general problem is in Π 2 p . However, we provide good news too. We identify a pertinent class of unfoldings for which the removal of such unfoldings always results in a simpler disassembled view than the original view itself. We also develop an algorithm that finds rewrites equivalent to the disassembled view which are, in a sense, locally optimal. The algorithm establishes a cover completion by finding a (minimal) collection of unfoldings of the view that, along with the unfoldings to be removed, covers the original view. This approach is effectively tractable, unlike the search for globally, or absolutely, optimal rewrites. Furthermore, we show that these cover-completion rewrites are preferable to absolutely optimal rewrites in many ways.

Reducing the cost of accessing relations in incremental view maintenance

In the data warehouse environment, the concept of a materialized view is common and important for efficient support of OLAP query processing. Materialized views are generally derived from several relations. These materialized views need to be updated when source relations change. Since the propagation of updates to the views may impose a significant overhead, it is essential to update the warehouse views efficiently. Though various view maintenance strategies have been discussed in the past, optimizations on the total accesses to relations have not been sufficiently investigated. In this paper we propose an efficient incremental view maintenance method called optimal delta evaluation that can minimize the total accesses to relations. We first present the delta evaluation expression and a delta evaluation tree which are core concepts of the method. Then, a dynamic programming algorithm that can find the optimal delta evaluation tree is proposed. We also present various experimental results that show the usefulness and efficiency of our proposed method.

View Maintenance Strategy in Peer Data Management Systems

In this paper, a strategy for view maintenance in peer data management systems (PDMSs) is proposed. First, a hybrid peer architecture is presented, in which peers prefer peer-to-peer architecture to super-peer based architecture. Then, if a view is reformulated along a semantic path, it can retrieve data from many data sources. So the peer view, local view and global view are proposed. A global view maintenance is turned into the maintenance of all related local views if join operations are confined in each local PDMS, and a query can gain the same result if it is posed over any peer in the same semantic path. Furthermore, a view may be a conjunctive query which relates to several tables in PDMSs. According to the application, Mork’s rules are extended. Based on the new rule system, the update data are propagated in PDMS. According to the new rule system, a view maintenance algorithm in PDMS is proposed. Finally, simulation experiments are carried out in the SPDMS (schema-mapping-based PDMS). The simulation results show that the proposed view maintenance strategy has better performance than that of Mork’s.

Parallel generation of base relation snapshots for materialised view maintenance in data warehouse environment

A data warehouse organises and stores data needed for informational, analytical processing over a long historical time perspective. These keep a materialised view (such as historical data), and user queries are processed using this view. The view has to be maintained to reflect the updates done against the base relations stored at the various distributed data sources. For unsophisticated sources, in practice it is often necessary to infer modifications by periodically comparing snapshots and backup copies of data from the source. This study considers the materialised view and its maintenance. Various differential snapshot algorithms have been compared.

Statistical Sampling to Instantiate Materialized View Selection Problems in Data Warehouses

In any online decision support system, the backbone is a data warehouse. In order to facilitate rapid response to complex business decision support queries, it is a common practice to materialize an appropriate set of the views at the data warehouse. However, it typically requires the solution of the Materialized View Selection (MVS) problem to select the right set of views to materialize in order to achieve a certain level of service given a limited amount of resource such as materialization time, storage space, or view maintenance time. Dynamic changes in the source data and the end users requirement necessitate rapid and repetitive instantiation and solution of the MVS problem. In an online decision support context, time is of the essence in finding acceptable solutions to this problem. In this chapter, we have used a novel approach to instantiate and solve four versions of the MVS problem using three sampling techniques and two databases. We compared these solutions with the optimal solutions corresponding to the actual problems. In our experimentation, we found that the sampling approach resulted in substantial savings in time while producing good solutions.

A compensation-based approach for view maintenance in distributed environments

Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored as materialized views to allow better access, performance, and high availability. In loosely coupled environments, such as the data grid, the data sources are autonomous. Hence, tie source updates can be concurrent and cause erroneous results during view maintenance. State-of-the-art maintenance strategies apply compensating queries to correct such errors, making the restricting assumption that all source schemata remain static over time. However, in such dynamic environments, the data sources may change not only their data but also their schema. Consequently, either the maintenance queres or the compensating queries may fail. In this paper, we propose a novel framework called DyDa that overcomes these limitations and handles both source data updates and schema changes. We identify three types of maintenance anomalies, caused by either source data updates, data-preserving schema changes, or non-data-preserving schema changes. We propose a compensation algorithm to solve the first two types of anomalies. We show that the third type of anomaly is caused by the violation of dependencies between maintenance processes. Then, we propose dependency detection and correction algorithms to identify and resolve the violations. Put together, DyDa extends prior maintenance solutions to solve all types of view maintenance anomalies. The experimental results show that DyDa imposes a minimal overhead on data update processing while allowing for the extended functionality to handle concurrent schema changes

2DCMA: An Effective Maintenance Algorithm of Materialized Views in Peer Data Management Systems

Update management is very important for data integration systems. So update management in peer data management systems (PDMSs) is a hot research area. This paper researches on view maintenance in PDMSs. First, the definition of view is extended and the peer view, local view and global view are proposed according to the requirements of applications. There are two main factors to influence materialized views in PDMSs. One is that schema mappings between peers are changed, and the other is that peers update their data. Based on the requirements, this paper proposes an algorithm called 2DCMA, which includes two sub-algorithms: data and definition consistency maintenance algorithms, to effectively maintain views. For data consistency maintenance, Mork’s rules are extended for governing the use of updategrams and boosters. The new rule system can be used to optimize the execution plan. And are extended for the data consistency maintenance algorithm is based on the new rule system. Furthermore, an ECA rule is adopted for definition consistency maintenance. Finally, extensive simulation experiments are conducted in SPDMS. The simulation results show that the 2DCMA algorithm has better performance than that of Mork’s when maintaining data consistency. And the 2DCMA algorithm has better performance than that of centralized view maintenance algorithm when maintaining definition consistency.

Maintaining large update batches by restructuring and grouping

Materialized views defined over distributed data sources can be utilized by many applications to ensure better access, reliable performance, and high availability. Technology for maintaining materialized views is thus critical for providing up-to-date results since a stale view extent may not help or even mislead these applications. State-of-the-art incremental view maintenance requires O ( n 2 ) or more remote maintenance queries with n being the number of data sources in the view definition. In this work, we propose two novel maintenance strategies, namely adjacent grouping and conditional grouping, that dramatically reduce the number of maintenance queries required to maintain the materialized views. This reduction in the number of maintenance queries brings the basic trade-off between the complexity of each query and the total number of maintenance queries that can be exploited to improve maintenance performance. The proposed maintenance strategies have been implemented in a working prototype system called TxnWrap. Experimental studies illustrate that our proposed strategies are able to achieve about 400% performance improvement in terms of total processing time compared with existing batch algorithms in a majority of cases.

Optimizing cyclic join view maintenance over distributed data sources

Materialized views defined over distributed data sources are critical for many applications to ensure efficient access, reliable performance, and high availability. Materialized views need to be maintained upon source updates since stale view extents may not serve well or may even mislead user applications. Thus, view maintenance performance is one of the keys to the success of these applications. In this work, we investigate two maintenance strategies, extended batching and view graph transformation, for maintaining general join views where join conditions may exist between any pairs of data sources possibly with cycles. Many choices are available for maintaining cyclic join views. We thus propose a cost-driven view maintenance framework which generates optimized maintenance plans tuned to the environmental settings. The proposed framework has been implemented in the TxnWrap system. Experimental studies illustrate that our proposed optimization techniques significantly improve the view maintenance performance in a distributed environment.

Incremental maintenance of shortest distance and transitive closure in first-order logic and SQL

Given a database, the view maintenance problem is concerned with the efficient computation of the new contents of a given view when updates to the database happen. We consider the view maintenance problem for the situation when the database contains a weighted graph and the view is either the transitive closure or the answer to the all-pairs shortest-distance problem ( APSD ). We give incremental algorithms for APSD , which support both edge insertions and deletions. For transitive closure, the algorithm is applicable to a more general class of graphs than those previously explored. Our algorithms use first-order queries, along with addition (+) and less-than (<) operations ( FO (+,<)); they store O ( n 2 ) number of tuples, where n is the number of vertices, and have AC 0 data complexity for integer weights. Since FO (+,<) is a sublanguage of SQL and is supported by almost all current database systems, our maintenance algorithms are more appropriate for database applications than nondatabase query types of maintenance algorithms.

Locking protocols for materialized aggregate join views

The maintenance of materialized aggregate join views is a well-studied problem. However, to date the published literature has largely ignored the issue of concurrency control. Clearly, immediate materialized view maintenance with transactional consistency, if enforced by generic concurrency control mechanisms, can result in low levels of concurrency and high rates of deadlock. While this problem is superficially amenable to well-known techniques, such as fine-granularity locking and special lock modes for updates that are associative and commutative, we show that these previous high concurrency locking techniques do not fully solve the problem, but a combination of a "value-based" latch pool and these previous high concurrency locking techniques can solve the problem.

Multiagent Immediate Incremental View Maintenance for Data Warehouses

Data warehouse systems typically designate downtime for view maintenance, ranging from tens of minutes to hours depending on the system size. We develop a multiagent system that achieves immediate incremental view maintenance (IIVM) for continuous updating of data warehouse views. We describe an IIVM system that processes updates as transactions are executed at the underlying data sources to eliminate view maintenance downtime for the data warehouse-a crucial requirement for internet applications. The use of a multiagent framework provides considerable process speed improvement when compared with other IIVM systems. Since agents are used to delegate the data sources and warehouse views, it is easy to reorganize the components of the system. Through the use of cooperative agents, the data consistency of IIVM can be easily maintained. The test results from this research show that the proposed system increases the availability of the data warehouse while preserving a stringent requirement on data consistency.

Popper’s Historical Role: Innovative Dissident

Whether Popper’s philosophy will be used widely enough to shape the philosophy of science in the future will determine what his role in the history of the philosophy of science will be. The choice is that between the quest for deeper understanding of science and society, on the one hand, and the maintenance of old and comfortable views, on the other. Although in the past comfort has normally won out, progress has been made by dissidents such as Maimon and Whewell. If the philosophy of science remains progressive, Popper will have the same kind of success.

Data warehouse clustering on the web

In collaborative e-commerce environments, interoperation is a prerequisite for data warehouses that are physically scattered along the value chain. Adopting system and information quality as success variables, we argue that what is required for data warehouse refreshment in this context is inherently more complex than the materialized view maintenance problem and we offer an approach that addresses refreshment in a federation of data warehouses. Defining a special kind of materialized views, we propose an open multi-agent architecture for their incremental maintenance while considering referential integrity constraints on source data.

Multiversion-based view maintenance over distributed data sources

Materialized views can be maintained by submitting maintenance queries to the data sources. However, the query results may be erroneous due to concurrent source updates. State-of-the-art maintenance strategies typically apply compensations to resolve such conflicts and assume all source schemata remain stable over time. In a loosely coupled dynamic environment, the sources may autonomously change not only their data but also their schema or semantics. Consequently, either the maintenance or the compensation queries may be broken. Unlike compensation-based approaches found in the literature, we instead model the complete materialized view maintenance process as a view maintenance transaction (VM_Transaction). This way, the anomaly problem can be rephrased as the serializability of VM_Transactions. To achieve VM_Transaction serializability, we propose a multiversion concurrency control algorithm, called TxnWrap , which is shown to be the appropriate design for loosely coupled environments with autonomous data sources. TxnWrap is complementary to the maintenance algorithms proposed in the literature, since it removes concurrency issues from consideration allowing the designer to focus on the maintenance logic. We show several optimizations of TxnWrap, in particular, (1) space optimizations on versioned data materialization and (2) parallel maintenance scheduling. With these optimizations, TxnWrap even outperforms state-of-the-art view maintenance solutions in terms of refresh time. Further, several design choices of TxnWrap are studied each having its respective advantages for certain environmental settings. A correctness proof based on transaction theory for TxnWrap is also provided. Last, we have implemented TxnWrap. The experimental results confirm that TxnWrap achieves predictable performance under a varying rate of concurrency.

Efficiently supporting order in XML query processing

XML is an ordered data model and XQuery expressions return results that have a well-defined order. However, little work on how order is supported in XML query processing has been done to date. In this paper we study the issues related to handling order in the XML context, namely challenges imposed by the XML data model, the variety of order requirements of the XQuery language, and the need to maintain order in the presence of updates to the XML data. We propose an efficient solution that addresses all these issues. Our solution is based on a key encoding for XML nodes that serves as node identity and at the same time encodes order. We design rules for encoding order of processed XML nodes based on the XML algebraic query execution model and the node key encoding. These rules do not require any actual sorting for intermediate results during execution. Our approach enables efficient order-sensitive incremental view maintenance as it makes most XML algebra operators distributive with respect to bag union. We prove the correctness of our order encoding approach. Our approach is implemented and integrated with Rainbow, an XML data management system developed at WPI. We have tested the efficiency of our approach using queries that have different order requirements. We have also measured the relative cost of different components related to our order solution in different types of queries. In general the overhead of maintaining order in our approach is very small relative to the query processing time.

Incremental maintenance of schema-restructuring views in SchemaSQL

The integration of data, especially from heterogeneous sources, is a hard and widely studied problem. One particularly challenging issue is the integration of sources that are semantically equivalent but schematically heterogeneous. While two such data sources may represent the same information, one may store the information inside tuples (data) while the other may store it in attribute or relation names (schema). The SchemaSQL query language is a recent solution to this problem powerful enough to restructure such sources into each other without the loss of information. We propose the first incremental view maintenance strategy for such schema-restructuring views. Our strategy, based on an algebraic representation of the view query, correctly transforms a data update or a schema change to a source into sequences of schema and data updates to be applied to the view. We also introduce an optimization of incremental maintenance using batching. We present a proof of correctness of the propagation approach. We also describe the implementation of our SchemaSQL Query Processor and View Maintainer. Last, our experimental results demonstrate that, in many cases, incremental SchemaSQL view maintenance is significantly faster than complete view recomputation.

Data Warehousing Interoperability for the Extended Enterprise

The purpose of this paper is to raise awareness and identify a number of challenges regarding the issue of data warehouse interoperation in Web-based collaborative environments. Adopting system and information quality as success variables, we argue that existing works fell short of addressing complex issues that relate to their refreshment and extent far beyond view maintenance solutions within single warehouses. Considering a solution that approaches warehouse refreshment as a business process in a federation of data warehouses, we define a special kind of materialized view that emanates from such an environment, provide a roadmap for implementing the appropriate warehousing architecture, and give some preliminary empirical results.

Conflict Order-Based View Refreshment Scheme for Transaction Management in Data Warehouse Environment

Online analytical processing transactions (OLAP for short) are executed in integrated a data warehouse (DWH for short). DWH should be refreshed by recent value of data in source databases in order to prevent OLAP transactions from reading excessively stale data. This refreshing process is referred to as view maintenance. There have been many researches on ways to guarantee data consistency during the process of view maintenance. The most commonly used approach among them is to execute updated transactions in the nighttime while OLAP transactions are executed only in daytime. It is, however, meaningless to argue about absolute nighttime for a specific corporation which is globalized. We proposed so called COB algorithm for view maintenance in DWH environment. COB can improve data freshness by supporting concurrent execution of one or more update transactions. COB can, moreover, reduce the cost of space management by maintaining only one additional version during the process of view maintenance.

Parallel multisource view maintenance

In a distributed environment, materialized views are used to integrate data from different information sources and then store them in some centralized location. In order to maintain such materialized views, maintenance queries need to be sent to information sources by the data warehouse management system. Due to the independence of the information sources and the data warehouse, concurrency issues are raised between the maintenance queries and the local update transactions at each information source. Recent solutions such as ECA and Strobe tackle such concurrent maintenance, however with the requirement of quiescence of the information sources. SWEEP and POSSE overcome this limitation by decomposing the global maintenance query into smaller subqueries to be sent to every information source and then performing conflict correction locally at the data warehouse. Note that all these previous approaches handle the data updates one at a time. Hence either some of the information sources or the data warehouse is likely to be idle during most of the maintenance process. In this paper, we propose that a set of updates should be maintained in parallel by several concurrent maintenance processes so that both the information sources as well as the warehouse would be utilized more fully throughout the maintenance process. This parallelism should then improve the overall maintenance performance. For this we have developed a parallel view maintenance algorithm, called PVM, that substantially improves upon the performance of previous maintenance approaches by handling a set of data updates at the same time. The parallel handling of a set of updates is orthogonal to the particular maintenance algorithm applied to the handling of each individual update. In order to perform parallel view maintenance, we have identified two critical issues that must be overcome: (1) detecting maintenance-concurrent data updates in a parallel mode and (2) correcting the problem that the data warehouse commit order may not correspond to the data warehouse update processing order due to parallel maintenance handling. In this work, we provide solutions to both issues. For the former, we insert a middle-layer timestamp assignment module for detecting maintenance-concurrent data updates without requiring any global clock synchronization. For the latter, we introduce the negative counter concept to solve the problem of variant orders of committing effects of data updates to the data warehouse. We provide a proof of the correctness of PVM that guarantees that our strategy indeed generates the correct final data warehouse state. We have implemented both SWEEP and PVM in our EVE data warehousing system. Our performance study demonstrates that a manyfold performance improvement is achieved by PVM over SWEEP.