Challenges of designing hand recognition for a manual assembly assistance system

Abstract

Manual assembly remains an important task in production with changing requirements for the worker due to i.e. mass personalized products. To keep up with these requirement changes, modern assistance systems are suitable to support the workers. The goals of these assistance systems are to detect errors in the process, guide the worker through new processes and document the process. To achieve this, the assistance system needs to follow each step of the worker reliably. This can be realized with a visual scene analysis based on machine learning. In the presented work, one 3D-sensor (active stereo vision with an additional RGB camera) is used for the scene analysis. The presented work evaluates existing methods for hand localization and hand pose recognition for the application in manual assembly assistance systems. A new procedure for hand localization based on current machine learning techniques is developed specifically for the scenario of recognizing hand joints (also called hand pose) in a manual assembly scenario. Additionally, different methods for hand pose recognition are compared in the application scenario. Based on the results of the developed hand localization method, current hand pose detection methods were evaluated (Dense-Regression and DeepPrior++). While the methods worked well in a scenario comparable to the dataset, they did not perform very well in a manual assembly scenario. Improvements can be made using pixel-wise segmentation or using specific datasets for training containing data from manual assembly scenarios.