Real-time Mixed Reality (MR) and Artificial Intelligence (AI) object recognition integration for digital twin in Industry 4.0

Abstract

In the digital era, real-time object recognition and inspection, based on Artificial Intelligence (AI), have become vitally important in a wide range of applications. With the evolution of Industry 4.0, the integration of digital informatics and decisions is crucial for smart manufacturing, Logistics 4.0, and supply chain digitization. However, much of digital content lacks interconnection. As such, the Digital Twin (DT) was proposed to promote the integration of physical machines and devices with the digital space. Augmented reality (AR) usually works in conjunction with the DT network to collect and provide real-time data. In fact, most of the AR devices are not fully connected in DT as the decision support from AI is disconnected. On the other hand, human-machines and object interactions are still complicated in AR, thus limiting applications. In order to achieve digital twinning for object recognition and human-machine interaction, we proposed a DT architecture for integrating the latest Mixed Reality (MR) device for real-time data streaming. Object recognition is empowered by the AI algorithm in the backend cloud computing and database. The entire system integrates the informatics between physical machines and devices and the digital content using an AI algorithm. Our proposed method achieved real-time informatics integration of MR and AI, while the warehouse is used as a case scenario with simulations. Industrial informatics integration can be applied for smart manufacturing, smart warehouse, and many Industry 4.0 applications.