An Augmented Reality-Based Method for Remote Collaborative Real-Time Assistance: from a System Perspective

Abstract

To provide remote assistance to people more efficiently, an augmented reality (AR)-based method for remote real-time assistance for collaboration is proposed. This paper aims to reduce communication barriers and enhance the three-dimensional (3D) feel of immersive interactions. First, a multiplayer real-time video communication framework with WebRTC is built, which enables remote experts to observe a first-hand view of an operator’s site. Second, a shared cross-platform virtual whiteboard based on Canvas, WebSocket and Node.js is developed that enables remote experts to provide visual assistance, such as drawings or text, and adjust the position of the whiteboard for seamless integration with video. Last, the virtual assistance information provided by the remote experts is displayed on the screen of AR holographic glasses to enhance the assistance capability of the platform and enable an expert to explain to an operator how to correctly perform tasks. A hybrid tracking and registration technique based on natural features and gyroscopes is adopted to estimate the operator’s posture in real time to enable the virtual assistance information to be perfectly integrated with the real world at all times. An experimental analysis shows that this system is both practicable and stable and has broad application prospects in many fields.