Container stacking optimization based on Deep Reinforcement Learning

Abstract

Cargo storage is one of the key aspects of the maritime transportation. As the prior site planning, container stacking has a critical influence on the operation efficiency of the storage yard. To store a group of containers in a certain number of stacks with capacity constraints in order, we propose a self-attention based Deep Reinforcement Learning (DRL) method, which can learn high-quality policy to solve the container stacking problem. We design a Markov Decision Process (MDP) model to simulate the container stacking process and enable the DRL agent to learn to minimize the number of blockages when retrieving the containers. In the proposed DRL model, a novel feature extraction network based on self-attention is utilized to effectively capture the interrelationships between stacks and represent the state of all stacks. Additionally, the size-agnostic policy network enables the agent to have the ability to handle problems of different scales. Through extensive experimental verification, our method significantly outperforms general stacking rules, heuristic-search algorithm and mathematical programming in medium-scale and large-scale problems. Specifically, the proposed DRL method outperforms the optimal existing method by 54% and 80% for medium and large-scale problems, respectively. Moreover, the learned policies exhibit outstanding generalization performance on unseen scenarios with different scales and settings.