E-ETL: Framework For Managing Evolving ETL Processes

Abstract

Data warehouses integrate external data sources (EDSs), which very often change their data structures (schemas). In many cases, such changes cause an erroneous execution of an already deployed ETL workflow. Structural changes of EDSs are frequent, therefore an automatic reparation of an ETL workflow, after such changes, is of a high importance. This paper presents a framework for handling the evolution of an ETL layer – E − ETL. Detection of changes in EDSs causes a reparation of the fragment of ETL workflow which interacts with the changed EDS. The proposed framework was developed as a module external to an ETL engine, accessing the engine by means of API. The innovation of this framework are algorithms for semi-automatic reparation of an ETL workflow.KeywordsData WarehouseEvolution RuleCommon Data ModelExternal Data SourceReparation AlgorithmThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.