Multiagent Q-Learning Approach for the Recharging Scheduling of Electric Automated Guided Vehicles in Container Terminals

Abstract

In recent years, advancements in battery technology have led to increased adoption of electric automated guided vehicles in container terminals. Given how critical these vehicles are to terminal operations, this trend requires efficient recharging scheduling for automated guided vehicles, and the main challenges arise from limited charging station capacity and tight vehicle schedules. Motivated by the dynamic nature of the problem, the recharging scheduling problem for an entire vehicle fleet given capacitated stations is formulated as a Markov decision process model. Then, it is solved using a multiagent Q-learning (MAQL) approach to produce a recharging schedule that minimizes the delay of jobs. Numerical experiments show that under a stochastic environment in terms of vehicle travel time, MAQL enables the exploration of better scheduling by coordinating across the entire vehicle fleet and charging facilities and outperforms various benchmark approaches, with an additional improvement of 18.8% on average over the best rule-based heuristic and 5.4% over the predetermined approach. Funding: This work was supported by the National Natural Science Foundation of China [Grant 72101203], the Shaanxi Provincial Key R&D Program, China [Grant 2022KW-02], and the Singapore Maritime Institute [Grant SMI-2017-SP-002]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2022.0113 .