Abstract
To provide a realistic environment for remote sensing applications, point clouds are used to realize a three-dimensional (3D) digital world for the user. Motion recognition of objects, e.g., humans, is required to provide realistic experiences in the 3D digital world. To recognize a user’s motions, 3D landmarks are provided by analyzing a 3D point cloud collected through a light detection and ranging (LiDAR) system or a red green blue (RGB) image collected visually. However, manual supervision is required to extract 3D landmarks as to whether they originate from the RGB image or the 3D point cloud. Thus, there is a need for a method for extracting 3D landmarks without manual supervision. Herein, an RGB image and a 3D point cloud are used to extract 3D landmarks. The 3D point cloud is utilized as the relative distance between a LiDAR and a user. Because it cannot contain all information the user’s entire body due to disparities, it cannot generate a dense depth image that provides the boundary of user’s body. Therefore, up-sampling is performed to increase the density of the depth image generated based on the 3D point cloud; the density depends on the 3D point cloud. This paper proposes a system for extracting 3D landmarks using 3D point clouds and RGB images without manual supervision. A depth image provides the boundary of a user’s motion and is generated by using 3D point cloud and RGB image collected by a LiDAR and an RGB camera, respectively. To extract 3D landmarks automatically, an encoder–decoder model is trained with the generated depth images, and the RGB images and 3D landmarks are extracted from these images with the trained encoder model. The method of extracting 3D landmarks using RGB depth (RGBD) images was verified experimentally, and 3D landmarks were extracted to evaluate the user’s motions with RGBD images. In this manner, landmarks could be extracted according to the user’s motions, rather than by extracting them using the RGB images. The depth images generated by the proposed method were 1.832 times denser than the up-sampling-based depth images generated with bilateral filtering.
Highlights
Three-dimensional (3D) digital environments are provided to take advantage of various platforms such as remotely controlled unmanned aerial vehicles and autonomous vehicles [1,2,3,4,5,6]
A depth image provides the boundary of the user’s motion, and the depth image is generated from the collected 3D point cloud and the difference image generated from the background image and collected red green blue (RGB) image
An improved encoder–decoder model is trained for extracting 3D landmarks using the generated user’s RGB depth (RGBD) images. 3D landmarks for user motions are extracted based on the trained encoder model and using the RGBD images
Summary
Three-dimensional (3D) digital environments are provided to take advantage of various platforms such as remotely controlled unmanned aerial vehicles and autonomous vehicles [1,2,3,4,5,6]. This paper proposes a system that automatically extracts 3D landmarks without manual supervision It uses RGB images and 3D point clouds collected with a vision system and a LiDAR, respectively, to recognize a user’s motions and does not depend on the number of points in the 3D point cloud. By utilizing the difference between the background RGB image and the RGB image that captures a user’s motion, the depth image of the user’s motion is generated by correcting the disparities in the 3D point cloud Based on these depth images, 3D landmarks of the user’s motions are automatically extracted with an improved encoder–decoder model; the 3D landmarks are generated by the trained encoder.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
