Design and implementation of a marker-based AR-enabled tool tracking system for manufacturing manual operation

Abstract

Although automation and robotics are widely implemented in manufacturing industry nowadays, assembly tasks and rework processes are still carried out manually by human operators because of their complexity and the need for a level of adaptability and flexibility greater than automation solutions can provide. However, manual operations exhibit variability and are subject to human errors, which could lead to unexpected delays or quality issues. Enabling traceability of manual operations is also intrinsically more challenging than for automated processes as it typically relies on the operator consistently providing direct input (e.g. HMI/operator interaction).Research suggests that augmented reality (AR) technology can contribute to enhance human-machine interaction by providing operators with a seamless digital bi-directional interface with physical systems (e.g. product or production systems), thus improving manual operations’ overall effectiveness. Existing research related to the development of AR-based solutions for manufacturing focus essentially on training and maintenance use cases, while there is limited development of applications aiming at supporting in-production operations. In addition, while AR technologies are used to provide information to the operator, the development of capabilities allowing manual process and operators to be monitored during operations, are lacking. The research presented in this paper adopt a holistic approach combining manual operation monitoring and operator feedback capabilities. The DAMPO (Digitally Augmented Manual Process Optimisation) system implements a) computer vision technologies to provide continuous operator and manual operations monitoring capabilities b) content rich and highly interactive user interfaces using screen-based 3D and AR-based information display, and c) near real-time data capture, management and processing pipelines that provide both real-time system/user interaction and collection of historical process data. The use case for this research focuses on specific manual assembly operations of safety critical components of seating systems for the automotive industry, which require precise sequence of operations and full traceability of the assembly process for audit purposes (safety critical operations). The DAMPO system is implemented in the Automation Systems Group's Digital Automation Laboratory at the University of Warwick, WMG department, UK, and is used to support both low TRL level research and direct engagement with industry on developing human centric manufacturing solutions. The DAMPO systems implement both process monitoring capabilities (i.e. data capture) and Electronic Work Instruction functions (information feedback and presentation of the operator) in view of improving process traceability and implementing no-fault forward capabilities. The DAMPO solutions combines a wide range of a) visual computing methods (fiducial, IR marker and 3D object based pose estimation), and b) information display and operator interface technologies (e.g. projected, screen-based and head-mounted AR layer display, HMI and work instruction screens with interactive 3D content, haptic feedback), which can be combined differently depending on the use case, and use case requirements such as hand free operations, complexity of work instructions or operator feedback and interactions, etc. This production-ready solution is tested and evaluated on a replicable production cell using real product and real assembly process.