Development of Real-time Assembly Work Monitoring System Based on 3D Skeletal Model of Arms and Fingers

Abstract

Decreasing birth rates and an aging population in society often cause labor shortages in the manufacturing industry, making the development of methods to improve productivity based on limited human resources imperative. One way to achieve this is by enhancing product quality via the reduction of product losses and the consequent need for reassembly due to human error. Human error during assembly can arise from specific actions via the arms and fingers. We assumed that these errors can be captured based on the information on human skeletal models. The purpose of this study is to propose and develop a system that acquires information about assembly procedures by using human skeletal models, including the fingers of workers, and notifies them of skipped procedures and errors in part types. Further, we confirm the basic capability of the proposed system via experiments based on simulated assembly work. The proposed system monitors workers' motion based on color and depth images captured by a single RGB-D camera. In addition, we developed a function to detect the process and to point out errors with audible and visual feedbacks when errors were made by workers in assembly processes. In the experiment using simulated assembly work, the proposed system exhibited an accuracy rate of 98.3% with respect to acquisition of assembly processes. In the case of an error in an assembly process, the system was able to point out the error correctly and provide feedback to the worker before he had finished picking up the wrong part. In conclusion, we confirmed that the developed system exhibited the basic capability to acquire work procedures and efficiently point out errors in real time.