The AI circular hydrogen economist: Hydrogen supply chain design via hierarchical deep multi-agent reinforcement learning

Abstract

Hydrogen supply chain (HSC) consists of various production, conditioning, transportation, storage, and distribution processes, all of which require extensive computational resources for precise modelling. In this context, artificial intelligence (AI) is emerging as a pivotal tool for addressing model-based decision-making challenges, courtesy of its rapid and efficient computational capabilities. This paper proposes a comprehensive HSC model consisting of an economic policy planner, a wholesale hydrogen market, a power supply system, a hydrogen distribution system (HDS), and hydrogen refueling stations (HRSs). It leverages an AI circular hydrogen economist approach based on a hierarchical deep multi-agent reinforcement learning (MARL) algorithm, offering a new alternative to traditional multi-objective bi-level optimization platforms. The model incorporates green, blue, and gray hydrogen production processes as viable hydrogen production pathways, with the hierarchical MARL’s agents representing the HDS and HRSs at two decision-making levels. Energy requirements are met through a combination of on-site renewable energy sources, the main power grid, or distributed power generation systems. Several HSC scenarios are examined with respect to various combinations of green hydrogen supply rates and carbon credits granted to them under optimum conditions. The results showed that the developed hierarchical MARL has the potential to replace mathematical programming (MP), uncovering a new economic-environmental trade-off between profit of HDS and operational costs of HRSs. Notably, green hydrogen transactions exponentially increase within the supply chain as the carbon credits exceed $1 per kilogram of hydrogen. While this research focuses on optimizing daily operations within the HSC, future efforts can aim to extend this optimization to forecasted annual operations.