Schema Evolution Research Articles

Automating the database schema evolution process

Supporting database schema evolution represents a long-standing challenge of practical and theoretical importance for modern information systems. In this paper, we describe techniques and systems for automating the critical tasks of migrating the database and rewriting the legacy applications. In addition to labor saving, the benefits delivered by these advances are many and include reliable prediction of outcome, minimization of downtime, system-produced documentation, and support for archiving, historical queries, and provenance. The PRISM/PRISM++ system delivers these benefits, by solving the difficult problem of automating the migration of databases and the rewriting of queries and updates. In this paper, we present the PRISM/PRISM++ system and the novel technology that made it possible. In particular, we focus on the difficult and previously unsolved problem of supporting legacy queries and updates under schema and integrity constraints evolution. The PRISM/PRISM++ approach consists in providing the users with a set of SQL-based Schema Modification Operators (SMOs), which describe how the tables in the old schema are modified into those in the new schema. In order to support updates, SMOs are extended with integrity constraints modification operators. By using recent results on schema mapping, the paper (i) characterizes the impact on integrity constraints of structural schema changes, (ii) devises representations that enable the rewriting of updates, and (iii) develop a unified approach for query and update rewriting under constraints. We complement the system with two novel tools: the first automatically collects and provides statistics on schema evolution histories, whereas the second derives equivalent sequences of SMOs from the migration scripts that were used for schema upgrades. These tools were used to produce an extensive testbed containing 15 evolution histories of scientific databases and web information systems, providing over 100 years of aggregate evolution histories and almost 2,000 schema evolution steps.

Automata-based Static Analysis of XML Document Adaptation

The structure of an XML document can be optionally specified by means of XML Schema, thus enabling the exploitation of structural information for efficient document handling. Upon schema evolution, or when exchanging documents among different collections exploiting related but not identical schemas, the need may arise of adapting a document, known to be valid for a given schema S, to a target schema S'. The adaptation may require knowledge of the element semantics and cannot always be automatically derived. In this paper, we present an automata-based method for the static analysis of user-defined XML document adaptations, expressed as sequences of XQuery Update update primitives. The key feature of the method is the use of an automatic inference method for extracting the type, expressed as a Hedge Automaton, of a sequence of document updates. The type is computed starting from the original schema S and from rewriting rules that formally define the operational semantics of a sequence of document updates. Type inclusion can then be used as conformance test w.r.t. the type extracted from the target schema S'.

Efficient adaptation of XML data using a conceptual model

One of the prominent characteristics of XML applications is their dynamic nature. Changes in user requirements cause changes in schemas used in the systems and changes in the schemas subsequently make existing documents invalid. In this work, we study two tightly coupled problems--schema evolution and document adaptation. The presented approach extends an existing conceptual model for evolution of XML applications towards document adaptation, by introducing a formal framework for detecting changes between two versions of a schema. From the detected changes it is possible to create a script that transforms documents valid against the old version of the schema to documents valid against its new version.

Automatically Extracting Embedded Schemas from Regular Applications

A RDBMS is usually accessed using blocking drivers like JDBC/ODBC which require clients to block and wait for the result of each query they issue. An asynchronous database access mechanism would eliminate the need for such blocking and greatly improve client performance. Thread-Per-Connection and Thread Pooling are two methods currently being used to provide this asynchrony. This is inefficient since a lot of memory and computing power is spent in creating, scheduling and switching multiple threads. By these methods we cannot show how asynchronous database access can be achieved with a single thread using the Fork-Join mechanism which employs Future objects in Java. So, in this paper we propose an approach for automatically extracting embedded schemas from regular applications, e.g., written in java and automatically computing how schemas change as applications evolve. To showcase our approach, we perform a long-term schema evolution study. Our platform can be used for performing long-term, large-scale embedded schema evolution studies that are potentially beneficial to dynamic updating and schema evolution researchers.

Class Schema Evolution for Persistent Object-Oriented Software: Model, Empirical Study, and Automated Support

With the wide support for object serialization in object-oriented programming languages, persistent objects have become common place and most large object-oriented software systems rely on extensive amounts of persistent data. Such systems also evolve over time. Retrieving previously persisted objects from classes whose schema has changed is however difficult, and may lead to invalidating the consistency of the application. The ESCHER framework addresses these issues through an IDE-integrated approach that handles class schema evolution by managing versions of the code and generating transformation functions automatically. The infrastructure also enforces class invariants to prevent the introduction of potentially corrupt objects. This article describes a model for class attribute changes, a measure for class evolution robustness, four empirical studies, and the design and implementation of the ESCHER system.

Measuring complexity in OGC web services XML schemas: pragmatic use and solutions

The use of standards in the geospatial domain, such as those defined by the Open Geospatial Consortium (OGC), for exchanging data has brought a great deal of interoperability upon which systems can be built in a reliable way. Unfortunately, these standards are becoming increasingly complex, making their implementation an arduous task. The use of appropriate software metrics can be very useful to quantify different properties of the standards that ultimately may suggest different solutions to deal with problems related to their complexity. In this regard, we present in this article an attempt to measure the complexity of the schemas associated with the OGC implementation specifications. We use a comprehensive set of metrics to provide a multidimensional view of this complexity. These metrics can be used to evaluate the impact of design decisions, study the evolution of schemas, and so on. We also present and evaluate different solutions that could be applied to overcome some of the problems associated with the complexity of the schemas.

Schema Evolution via Multi-Version Metadata in SaaS

The Basic-Table combined with Extension-Table (BT&ET) layout has become the popular data storage architecture for SaaS currently. For the sake of improving the processing efficiency we improve the BT&ET layout, and migrate some tenants’ frequently accessed extension fields into the basic table based on the tenants’ constantly need on data access. With the development of cloud computing, Multi-Tenant data need to be stored in multiple data nodes. When tenant's data storage schema evolution happens, all data nodes hava to do data migration simultaneously and tenants may perceive the influence. In order to minimize the costs and negative load of schema evolution we propose the Multi-Version metadata technology. Tenant's data on each data node may contain different version metadata, and schema envolution can run asynchronously. We carry out experiments and the results figure out that the workload decreases significantly.

Special issue on business process lifecycle: design, deployment, operation and evaluation

Keywords: Business Process ; Lifecycle ; Deployment Reference EPFL-ARTICLE-176301doi:10.1002/smr.559View record in Web of Science Record created on 2012-04-19, modified on 2017-05-12

Toward enhanced life‐cycle support for declarative processes

SUMMARYThe need for flexible process‐aware information systems resulted in a recent interest in declarative approaches, as they promise a high degree of flexibility. However, the potential of current declarative approaches is impeded by deficiencies in understandability and maintainability. This paper proposes an approach toward better understandability and maintainability of declarative processes by adopting well‐established techniques from the domain of software engineering. More specifically, the ideas of test‐driven development and automated acceptance testing are adopted to interweave process specification and process testing. Thereby, during modeling, testcases balance the circumstantial/sequential information mismatch as well as improve understandability by dispensing with hard mental operations and removing hidden dependencies. Because testcases are also understandable to domain experts, they foster communication between domain experts and model builders, providing a common basis for communication. During process execution, testcases, in turn, help to document the reasons for process deviations and ensure that respective deviations can be easily considered during schema evolution. Furthermore, testcases ensure that no undesired behavior is introduced through process adaptations. Copyright © 2011 John Wiley & Sons, Ltd.

Impact of XML Schema Evolution

We consider the problem of XML Schema evolution. In the ever-changing context of the web, XML schemas continuously change in order to cope with the natural evolution of the entities they describe. Schema changes have important consequences. First, existing documents valid with respect to the original schema are no longer guaranteed to fulfill the constraints described by the evolved schema. Second, the evolution also impacts programs, manipulating documents whose structure is described by the original schema. We propose a unifying framework for determining the effects of XML Schema evolution both on the validity of documents and on queries. The system is very powerful in analyzing various scenarios in which forward/backward compatibility of schemas is broken, and in which the result of a query may no longer be what was expected. Specifically, the system offers a predicate language that allows one to formulate properties related to schema evolution. The system then relies on exact reasoning techniques to perform a fine-grained analysis. This yields either a formal proof of the property or a counter-example that can be used for debugging purposes. The system has been fully implemented and tested with real-world use cases, in particular with the main standard document formats used on the web, as defined by W3C. The system precisely identifies compatibility relations between document formats. In case these relations do not hold, the system can identify queries that must be reformulated in order to produce the expected results across successive schema versions.

Schema Evolution for Data Warehouse: A Survey

warehouse is considered as the core component of the modern decision support systems. Due to the major support of data warehouse in the daily transaction of an enterprise, the requirements for the design and the implementation of DW are dynamic and subjective. This dynamic nature of the data warehouse may reflect the evolution in the data warehouse. Data warehouse evolution may be focused on three approaches namely schema evolution, schema versioning and view maintenance. Evolution of the data warehouse may often change their data and structure (schema changes). These schema changes may be consider according to the change in structure, software and users requirement. Schema evolution in data warehouse consists of various level namely structural level, conceptual level and behavioural level. This paper mainly focuses on schema evolution and proposes the operators to handle the creation and evolution of aggregated fact table. Our work is to do comparative study for various approaches of schema evolution.

Evolving schemas for streaming XML

In this paper we model schema evolution for XML by defining formal language operators on Visibly Pushdown Languages (VPLs). Our goal is to provide a framework for efficient validation of streaming XML in the realistic setting where the schemas of the exchanging parties evolve and thus diverge from one another. We show that Visibly Pushdown Languages are closed under the defined language operators and this enables us to expand the schemas (for XML) in order to account for flexible or constrained evolution.

Inconsistency-Tolerant Integrity Checking

All methods for efficient integrity checking require all integrity constraints to be totally satisfied, before any update is executed. However, a certain amount of inconsistency is the rule, rather than the exception in databases. In this paper, we close the gap between theory and practice of integrity checking, i.e., between the unrealistic theoretical requirement of total integrity and the practical need for inconsistency tolerance, which we define for integrity checking methods. We show that most of them can still be used to check whether updates preserve integrity, even if the current state is inconsistent. Inconsistency-tolerant integrity checking proves beneficial both for integrity preservation and query answering. Also, we show that it is useful for view updating, repairs, schema evolution, and other applications.

On Utilizing Web Service Equivalence for Supporting the Composition Life Cycle

Deciding on Web service equivalence in process-aware service compositions is a crucial challenge throughout the composition life cycle. However, restricting such decisions to (activity) label equivalence is not sufficient for many practical applications: if two activities and Web services respectively have equivalent labels, does this necessarily mean they are equivalent as well? In many scenarios (e.g., evolution of a composition schema or mining of completed composition instances), other factors also play an important role. Examples include context information (e.g., input and output messages) and information on the position of Web services within compositions. In this paper, the authors introduce the whole composition life cycle and discuss specific requirements for Web service equivalence along its different phases. The authors define adequate equivalence notions for the design, execution, analysis, and evolution of service compositions. This paper focuses on attribute and position equivalence and contributes a new understanding and treatment of equivalence notions in service compositions.

Clinical data integration of distributed data sources using Health Level Seven (HL7) v3-RIM mapping

BackgroundHealth information exchange and health information integration has become one of the top priorities for healthcare systems across institutions and hospitals. Most organizations and establishments implement health information exchange and integration in order to support meaningful information retrieval among their disparate healthcare systems. The challenges that prevent efficient health information integration for heterogeneous data sources are the lack of a common standard to support mapping across distributed data sources and the numerous and diverse healthcare domains. Health Level Seven (HL7) is a standards development organization which creates standards, but is itself not the standard. They create the Reference Information Model. RIM is developed by HL7's technical committees. It is a standardized abstract representation of HL7 data across all the domains of health care. In this article, we aim to present a design and a prototype implementation of HL7 v3-RIM mapping for information integration of distributed clinical data sources. The implementation enables the user to retrieve and search information that has been integrated using HL7 v3-RIM technology from disparate health care systems.Method and resultsWe designed and developed a prototype implementation of HL7 v3-RIM mapping function to integrate distributed clinical data sources using R-MIM classes from HL7 v3-RIM as a global view along with a collaborative centralized web-based mapping tool to tackle the evolution of both global and local schemas. Our prototype was implemented and integrated with a Clinical Database management Systems CDMS as a plug-in module. We tested the prototype system with some use case scenarios for distributed clinical data sources across several legacy CDMS. The results have been effective in improving information delivery, completing tasks that would have been otherwise difficult to accomplish, and reducing the time required to finish tasks which are used in collaborative information retrieval and sharing with other systems.ConclusionsWe created a prototype implementation of HL7 v3-RIM mapping for information integration between distributed clinical data sources to promote collaborative healthcare and translational research. The prototype has effectively and efficiently ensured the accuracy of the information and knowledge extractions for systems that have been integrated

Update rewriting and integrity constraint maintenance in a schema evolution support system

Supporting legacy applications when the database schema evolves represents a long-standing challenge of practical and theoretical importance. Recent work has produced algorithms and systems that automate the process of data migration and query adaptation; however, the problems of evolving integrity constraints and supporting legacy updates under schema and integrity constraints evolution are significantly more difficult and have thus far remained unsolved. In this paper, we address this issue by introducing a formal evolution model for the database schema structure and its integrity constraints, and use it to derive update mapping techniques akin to the rewriting techniques used for queries. Thus, we (i) propose a new set of Integrity Constraints Modification Operators (ICMOs), (ii) characterize the impact on integrity constraints of structural schema changes, (iii) devise representations that enable the rewriting of updates, and (iv) develop a unified approach for query and update rewriting under constraints. We then describe the implementation of these techniques provided by our PRISM++ system. The effectiveness of PRISM++ and its enabling technology has been verified on a testbed containing evolution histories of several scientific databases and web information systems, including the Genetic DB Ensembl (410+ schema versions in 9 years), and Wikipedia (240+ schema versions in 6 years).

Support for Schema Evolution in Data Stream Management Systems

Support for Schema Evolution in Data Stream Management Systems

Anchor modeling — Agile information modeling in evolving data environments

Maintaining and evolving data warehouses is a complex, error prone, and time consuming activity. The main reason for this state of affairs is that the environment of a data warehouse is in constant change, while the warehouse itself needs to provide a stable and consistent interface to information spanning extended periods of time. In this article, we propose an agile information modeling technique, called Anchor Modeling, that offers non-destructive extensibility mechanisms, thereby enabling robust and flexible management of changes. A key benefit of Anchor Modeling is that changes in a data warehouse environment only require extensions, not modifications, to the data warehouse. Such changes, therefore, do not require immediate modifications of existing applications, since all previous versions of the database schema are available as subsets of the current schema. Anchor Modeling decouples the evolution and application of a database, which when building a data warehouse enables shrinking of the initial project scope. While data models were previously made to capture every facet of a domain in a single phase of development, in Anchor Modeling fragments can be iteratively modeled and applied. We provide a formal and technology independent definition of anchor models and show how anchor models can be realized as relational databases together with examples of schema evolution. We also investigate performance through a number of lab experiments, which indicate that under certain conditions anchor databases perform substantially better than databases constructed using traditional modeling techniques.

A unit test approach for database schema evolution

Context The constant changes in today’s business requirements demand continuous database revisions. Hence, database structures, not unlike software applications, deteriorate during their lifespan and thus require refactoring in order to achieve a longer life span. Although unit tests support changes to application programs and refactoring, there is currently a lack of testing strategies for database schema evolution. Objective This work examines the challenges for database schema evolution and explores the possibility of using various testing strategies to assist with schema evolution. Specifically, the work proposes a novel unit test approach for the application code that accesses databases with the objective of proactively evaluating the code against the altered database. Method The approach was validated through the implementation of a testing framework in conjunction with a sample application and a relatively simple database schema. Although the database schema in this study was simple, it was nevertheless able to demonstrate the advantages of the proposed approach. Results After changes in the database schema, the proposed approach found all SELECT statements as well as the majority of other statements requiring modifications in the application code. Due to its efficiency with SELECT statements, the proposed approach is expected to be more successful with database warehouse applications where SELECT statements are dominant. Conclusion The unit test approach that accesses databases has proven to be successful in evaluating the application code against the evolved database. In particular, the approach is simple and straightforward to implement, which makes it easily adoptable in practice.

CODS

Database evolution is the process of updating the schema of a database or data warehouse (schema evolution) and evolving the data to the updated schema (data evolution). Database evolution is often necessitated in relational databases due to the changes of data or workload, the suboptimal initial schema design, or the availability of new knowledge of the database. It involves two steps: updating the database schema, and evolving the data to the new schema. Despite the capability of commercial RDBMSs to well optimize query processing, evolving the data during a database evolution through SQL queries is shown to be prohibitively costly. We designed and developed CODS, a platform for efficient data level data evolution in column oriented databases, which evolves the data to the new schema without materializing query results or unnecessary compression/decompression as occurred in traditional query level approaches. CODS ameliorates the efficiency of data evolution by orders of magnitude compared with commercial or open source RDBMSs.