Deep reinforcement learning with light-weight vision model for sequential robotic object sorting

Abstract

Training an agent to generate cooperative joint learning policies for executing object sorting in cluttered scenes is complex. This paper presents a comparative model-free deep reinforcement learning (DRL) system built from variants of end-to-end lightweight deep neural networks that jointly complement three primitive policies (pushing, grasping, and placing) that operate from visual observations to actions. The end goal of the proposed DRL systems is to perform object sorting in very dynamic and complex environmental scenarios of cluttered and random objects (irregular and regular blocks) existing in different color variations. For this, we investigated DRL methods built from 12 custom instances of Pixel-wise Q-valued Critic Networks (PQCN) from four main backbone networks (DenseNet121, DenseNet169, MobileNetV3, and SqueezeNet) individually combined with custom fully convolutional neural networks (FCN) for learning the image-based observation space through affordance mapping paradigm while considering both dual and single transfer learning schemes. Additionally, we explored three categories of gradient-based optimization methods and considered custom reward functions with varying discount factors. The results show that the PQCN constructed by integrating DenseNet121 or MobileNetV3 with FCN and trained by a dual transfer learning scheme (with a discount factor of γ=0.70) yielded the best performance in most categories of our evaluation in the training phases. Overall, the PQCN-DenseNet121, when trained with a dual transfer learning process, yielded the best generalization (sorting success rate) in most of the evaluation metrics across all object categories in the testing phase. Furthermore, our study provides a new benchmark for validating deep reinforcement learning algorithms while sorting object blocks under varying degrees of complexity. This research finds practical relevance in industrial fields such as manufacturing and packaging, building construction, plant sorting, automobile assembly lines, and waste sorting. Here is the article’s source code: https://github.com/Emmanuel-Okafor/DRL_for_Object_Sorting.git.