A novel design for Data Processing Framework of Park-level Power System with Data Mesh concept

Abstract

Building a new-type power system with renewable energy is the key to achieving the target of carbon peak and carbon neutrality, which has been the consensus of the clean, low-carbon, and safe energy transition. The construction of a new-type power system at the industrial park level is the first demonstration of the coordinated optimization and intelligent control of power sources, grid, load, and energy storage to make overall planning to achieve diversified development. To sustain the ‘balance, security, diversity, and low cost’ construction goals of the new-type power system at the park level, the power grid dispatching and control system is transformed to the direction of intelligence and automation, which is reflected in the rapid emergence and construction of complex systems such as digital twin, simulation, reinforcement learning, model training and prediction, and intelligent decision-making of the EMS (Energy Manage System). This brings more complex data sharing, interaction, construction, and other technical requirements than ever before. At the same time, along with the simultaneous construction of new power systems such as micro-grids, virtual power plants, and distributed photovoltaic systems, it has exacerbated the problems of data divergence from business, low data utilization, and centralized architecture brought by the data lakes and data warehouses under the previous monolithic architecture, leading to the data model hard to deploy to meet the demand for efficient collection, interaction, and sharing of scattered and heterogeneous data under the new generation energy structure and greatly increases the work pressure of the data team, making the data team a bottleneck for business applications. Therefore, this paper proposes a novel design scheme of data processing framework with the data mesh concept, supports the federal computing and analysis engine by building a distributed metadata center, changes the previous centralized data modeling method through data service and productization, realizes data sharing and interaction between complex multi-systems by defining service invocation and encapsulation technology means, improves data utilization, and better supports the automation, rationalization, and balanced construction of the intelligent dispatch system to meet the challenges and requirements brought by the new-type park-level power system.