Multiimaging Sensor Data Fusion-Based Enhancement for 3D Workspace Representation for Remote Machine Operation

Abstract

In incompletely characterized environments such as construction sites, remote machine operation is the preferred—and sometimes the only—safe and efficient solution for the operation of construction machines. When it comes to the operation of remote-controlled construction machines, a human—machine interface is needed so that even in the case of an unstructured environment (such as a construction site), the operator can interact with the machine in a safe and efficient manner. The human—machine interface needs to have the capability of realistically representing a three-dimensional (3D) workspace that provides information feedback to the remote operator. Workspace representation methods that are currently in use have certain limitations—they are time consuming and labor intensive and require high-performance computers. A major objective of this study is the development of an efficient means of representing a workspace in 3D that has the capacity to provide interactive visual feedback to the operator of remote-controlled construction machines. To achieve this objective, the ability is required to acquire dense, accurate, and visually realistic 3D data that can be converted into high-quality models. This allows creation of a realistic 3D workspace representation of terrain, including objects that might be in proximity to the machine. For this purpose, this study proposes a multiimaging sensor data fusion-based system employing joint bilateral upsampling, which enhances the quality and availability of the information acquired in such an environment. The field experiment results show that combining data acquired with a complementary multiimaging sensor setup allows enhancement of the quality of the information available to the remote operator. The resulting task-specific 3D workspace representation can be successfully incorporated into the development of remote-controlled construction machines that require interactive visual feedback. This provides the opportunity to make human—machine interaction more efficient and to improve the remote operation capability by assisting the operator. Moreover, the proposed multiimaging sensor data fusion-based 3D representation approach has the potential for effective use in a broad class of applications within the construction industry.