DOD-ETL: distributed on-demand ETL for near real-time business intelligence

Gustavo V Machado,Adriano C M Pereira,Ítalo Cunha,Leonardo B Oliveira

doi:10.1186/s13174-019-0121-z

Gustavo V Machado, Adriano C M Pereira + Show 2 more

Open Access

https://doi.org/10.1186/s13174-019-0121-z

Copy DOI

Abstract

The competitive dynamics of the globalized market demand information on the internal and external reality of corporations. Information is a precious asset and is responsible for establishing key advantages to enable companies to maintain their leadership. However, reliable, rich information is no longer the only goal. The time frame to extract information from data determines its usefulness. This work proposes DOD-ETL, a tool that addresses, in an innovative manner, the main bottleneck in Business Intelligence solutions, the Extract Transform Load process (ETL), providing it in near real-time. DOD-ETL achieves this by combining an on-demand data stream pipeline with a distributed, parallel and technology-independent architecture with in-memory caching and efficient data partitioning. We compared DOD-ETL with other Stream Processing frameworks used to perform near real-time ETL and found DOD-ETL executes workloads up to 10 times faster. We have deployed it in a large steelworks as a replacement for its previous ETL solution, enabling near real-time reports previously unavailable.

Highlights

Today, there is a dire need for organizations to find new ways to succeed in an increasingly competitive market
We proposed Distibuted On-Demand (DOD)-Extract Transform Load process (ETL), Distributed On-Demand ETL, a technology independent stack that combines multiple strategies to achieve near real-time ETL
DOD-ETL vs. previous works: improvements and trade-offs We identified that it is imperative for any near realtime ETL solution to have three key features [4]: high availability, low latency, and horizontal scalability

Summary

Introduction

There is a dire need for organizations to find new ways to succeed in an increasingly competitive market. Business Intelligence (BI) is a term used to define a variety of analytical tools that provide easy access to information that supports decision-making processes [1]. These tools perform collection, consolidation, and analysis of information, enabling analytical capabilities at every level inside and outside a company. Putting it another way, BI allows collected data to be unified, structured, and presented in an intuitive and concise manner, assisting organizations in corporate decision-making

Methods

Results

Conclusion