An Approach to Progress Monitoring of Industrial Manual Processes Based on Camera Recordings and Object Interactions

Abstract

Despite digitalization and automation, manual processes in factory operations still play a significant role. With the increasing complexity of tasks, it is advisable to support the employee during execution. Technological progress regarding the analysis of image data facilitates the use of camera-based systems for assistance functions. A core element for these assistance systems is the recognition of state or actions to achieve context-sensitive support.However, existing solutions are often developed for specific problems and are difficult to transfer to other processes. This is especially the case for manual processes with more and different sub-process steps such as setup and maintenance operations. High development effort even in case of minor process changes is the result. Therefore, the objective of this work is to develop a generic approach for progress monitoring of manual processes.The proposed approach is based on the understanding that manual processes can be considered as the manipulation of objects to achieve a defined state of their characteristic properties. Consequently, manual processes can be described by their features, like the positions or movements of objects or their relative position to each other. To realize a configurable approach a two-step procedure is proposed. The basis is the flexible recognition and localization of objects in camera recordings by machine learning methods. Subsequently, the information obtained can be utilized by a configurable logic to verify defined states.To develop the detailed concept, firstly, a method for analysis of complex manual processes in factory operation is conducted. The aim is to determine and categorize the objects relevant for optical recognition and state verification. Secondly, different machine learning approaches for object recognition and object localization are compared to evaluate their capability in capturing the feature states. The possible outputs of the machine learning approaches are used to create control elements for progress verification. Examples are the position of objects in relation to each other or the speed of movement. In the next step, a procedure for the process-specific selection of the control elements is developed. Finally, the approach is applied and discussed. A setup process of a turning machine and a deburring process serve as representative use cases.