Curriculum Learning in Peristaltic Sortation Machine

Abstract

This paper presents a novel approach to train a Reinforcement Learning (RL) agent faster for transportation of parcels in a Peristaltic Sortation Machine (PSM) using curriculum learning (CL). The PSM was developed as a means to transport parcels using an actuator and a flexible film where a RL agent is trained to control the actuator. In a previous paper, training of the actuator was done on a Discrete Element Method (DEM) simulation environment of the PSM developed using an open-source DEM library called LIGGGHTS, which reduced the training time of the transportation task compared to the real machine. But it still took days to train the agent. The objective of this paper is to reduce the training time to hours. To overcome this problem, we developed a faster but lower fidelity python simulation environment (PSE) capable of simulating the transportation task of PSM. And we used it with a curriculum learning approach to accelerate training the agent in the transportation process. The RL agent is trained in two steps in the PSE: 1. with a fixed set of goal positions, 2. with randomized goal positions. Additionally, we also use Gradient Monitoring (GM), a gradient regularization method, which provides additional trust region constraints in the policy updates of the RL agent when switching between tasks. The agent so trained is then deployed and tested in the DEM environment where the agent has not been trained before. The results obtained show that the RL agent trained using CL and PSE successfully completes the tasks in the DEM environment without any loss in performance, while using only a fraction of the training time (1.87%) per episode. This will allow for faster prototyping of algorithms to be tested on the PSM in future.