Real-time data warehousing in the big data environment: A comprehensive review of implementation in the internet industry

Abstract

Listen

Translate article

A real-time data warehouse is a crucial tool for information management and analysis, enabling the capture, processing, and analysis of vast amounts of data from diverse sources in real-time. It offers enterprises enhanced decision support through its efficient processing capabilities and timely data feedback. This paper reviews the technical characteristics and application scenarios of real-time data warehouses, with a particular focus on the Internet sector. It explores the evolution from traditional data warehouses to modern data lake and lakehouse architectures, emphasizing the advancements in data processing capabilities, including the separation of storage and compute functions. Real-time data warehouses, which enable immediate data processing and feedback, are essential for enterprises requiring up-to-the-minute insights. The study compares the Lambda and Kappa architectures, detailing their strengths and weaknesses in terms of data throughput, latency, and scalability. Innovations such as Apache Hudi and lakehouse architectures offer new opportunities for performance optimization and functional expansion. The emergence of hybrid architectures like HTAP (Hybrid Transactional/Analytical Processing) and HSAP (Hybrid Serving/Analytical Processing) represents a significant advancement in integrating transactional and analytical processing. Future research should focus on the impact of artificial intelligence and machine learning on real-time data warehouses to enhance their analytical and predictive capabilities, reduce complexity, and lower operational costs.