Mitigation of the Microsoft HoloLens’ hardware limitations for a controlled product assembly process

Abstract

Studies have shown that using augmented reality (AR) reduces time and errors by as much as 50% for manufacturing and assembly operations. The use of head-mounted displays (HMDs) is very interesting for these situations as they free up a user’s hands. However, with the technology behind these HMDs still under development, limitations in input, field of view, tracking, and occlusion exist. For AR assembly instructions to be effective, these limitations must be mitigated so that a user receives accurate instructions. To investigate this, an application was developed that guides a user through the assembly of a mock aircraft wing using AR on a Microsoft HoloLens 1. The application displays 3D work instructions, user interfaces, and spatially registered content. To ensure accurate instructions were displayed, techniques were developed to mitigate the HoloLens’ tracking and display limitations. Image tracking was implemented to augment the HoloLens’ limited position determination, stabilize spatially registered virtual content, and account for spatial drift. Also, 3D augmented content was optimized for clarity through the display using specialized computer graphics methods. Lastly, the system provides raw data regarding head position, orientation, and assembly step times. A visualization application was developed that combines this information with wearable sensor data, to examine trends exhibited by a user during the assembly tasks and validate the delivered instructions. The outcomes of this research provided solutions to address limitations of the HoloLens for broad use.