Real-time 3D unstructured environment reconstruction utilizing VR and Kinect-based immersive teleoperation for agricultural field robots

Abstract

Moving and operating autonomously in a field or orchard environment is challenging for agricultural robots due to the complex task requirements and highly unstructured conditions. The human–computer interaction-based remote control can provide the robots with an alternative solution to assist decision making and motion planning. In this study, a virtual reality (VR) and Kinect-based immersive teleoperation system were proposed to connect the physical and virtual world by utilizing real-time large-scale unstructured agricultural environment reconstruction and simultaneous virtual environment creation. The proposed system, with a relatively large server, can convert the scene into a realistic model by combing the depth and color image streams received from Kinect, and project them back into 3D (three dimensions) space in such a manner that the real 3D scene inside the camera’s field of view is recreated virtually. To create a VR environment for a VR headset, an optimized Bundlefusion-based algorithm was developed for real-time 3D reconstruction of the unstructured agricultural scene in the natural environment. Additionally, the performance of the proposed real-time 3D reconstruction algorithm was evaluated and compared with Bundlefusion and voxel hashing algorithms in different large-scale unstructured agricultural environments. Performing optimizing pose on our optimized algorithm leverages a large number of processing cores available to minimize the delay between data capture and rendering, and it reduces the average acquired time to process each frame no more than 0.9 ms. The experimental results including, lesscomputer storage occupied, fast frame processing time, and high-quality 3D realistic model indicate that our proposed system and algorithm have the potential applicability of immersive teleoperation in an unstructured agricultural environment.