Markerless cooperative augmented reality-based smart manufacturing double-check system: Case of safe PCBA inspection following automatic optical inspection

Abstract

Augmented reality (AR) is a key technology anchored towards realizing Industry 4.0 smart manufacturing aims. In manufacturing inspection, AR has previously been employed to support operators in assessing thickness of manufactured parts, ship building, aircraft subassemblies and printed circuit board assemblies (PCBAs) with or without external markers in standalone head-mounted (HMD) or handheld systems. These AR systems often use optical see-through (OST) or video see-through (VST) technologies. Currently, cyber physical integration of processes in an industrial system synergizes production through increased efficiency and improved quality while facilitating customization. An operator manually double-checking/inspecting a product that has previously been automatically inspected can better rely on this existing defect location information once it is contextually and spatially overlaid within their field of view to intuitively and efficiently execute the task. Additional interactive information provided can quickly reorient a user, provide necessary contextual reference, and monitor progress of the task while alerting them of their safety. In this study, adopting an automatic optical inspection (AOI) wirelessly aided HMD-OST AR-based manual inspection system requiring no external markers for contextual registration is a promising direction towards smart and safe PCBA manufacturing in line with Industry 4.0. Animated rectangular bracket of high color contrast is employed to localize the spawned AOI defect point, an arrow overlaid with distance text provides user guidance, a contextual resizable image facilitates user double-checking and a progress bar tracks the inspection progress while monitoring tracking state. We evaluate the developed system's robustness, effectiveness of display technology used, suitability of contextual against static display modes, and display technology influence on various inspection attributes in a user study. Results demonstrate the system to be robust, OST-HMD outperforms handheld VST devices, contextual display mode to be significantly preferred to static mode, and display technology employed has no significant influence on the inspection attributes. Finally, registration precision results demonstrate usability of the system while superposition of distance to orientation information raises the inspection rate of PCBAs.