Anomaly triplet-net: progress recognition model using deep metric learning considering occlusion for manual assembly work

Abstract

In this paper, we propose a method for recognizing progress in product assembly considering occlusion using deep metric learning. Visualizing the assembly process in factories is crucial for enhancing work efficiency and minimizing disposal costs. However, there is a problem in that products are managed by pasting them on paper with progress status written on them. We solve the problem of having to manage progress manually. First, the target assembly product is detected from images acquired from a fixed-point camera installed in the factory using a deep learning-based object detection method. Next, the detection area is cropped from the image. Finally, by using a classification method based on deep metric learning on the cropped image, the progress of the product assembly work is estimated as a rough progress step. As a specific progress estimation model, we propose an Anomaly Triplet-Net which is an improved model of the existing method Triplet Loss. This model considers occlusion by learning anomaly samples. In experiments, an 82.9 [%] success rate is achieved for the progress estimation method using Anomaly Triplet-Net. We also experimented with the practicality of the sequence of detection, cropping, and progression estimation, and confirmed the effectiveness of the overall system.