Research on Construction of Offline Data Warehouse for Ship Shore Power based on DolphinScheduler and Hive

Abstract

In today's networked, intelligent, and data-driven era, the shore power industry is facing the challenge of rapidly growing data. This paper presents the construction of a professional offline data warehouse system for shore power based on DolphinScheduler and Hive. Firstly, MySQL is adopted as the backend database, combined with Sqoop to synchronize business data to HDFS, ensuring data reliability and integrity. Secondly, the Flume-Kafka-Flume architecture is utilized to achieve real-time collection and caching of user behavior data, providing data support for subsequent analysis. Thirdly, HQL statements are written in Hive to clean, merge, and analyze shore power data, calculating key indicators such as electricity consumption and usage trends. Fourthly, data visualization is achieved through the integration of Superset, displaying data analysis results via a web interface. Fifthly, DolphinScheduler is employed for timed scheduling, ensuring dependency control among various tasks and the smooth operation of the project. This system fully leverages the replication mechanism of HDFS to enhance reliability, dynamically adds nodes to achieve system scalability, and fully utilizes the fault tolerance of the Yarn scheduler. It saves time and computational costs for the shore power industry, realizing higher value and benefits.