3D Depth Data Research Articles

N-DEPTH: Neural Depth Encoding for Compression-Resilient 3D Video Streaming

Recent advancements in 3D data capture have enabled the real-time acquisition of high-resolution 3D range data, even in mobile devices. However, this type of high bit-depth data remains difficult to efficiently transmit over a standard broadband connection. The most successful techniques for tackling this data problem thus far have been image-based depth encoding schemes that leverage modern image and video codecs. To our knowledge, no published work has directly optimized the end-to-end losses of a depth encoding scheme sandwiched around a lossy image compression codec. We present N-DEPTH, a compression-resilient neural depth encoding method that leverages deep learning to efficiently encode depth maps into 24-bit RGB representations that minimize end-to-end depth reconstruction errors when compressed with JPEG. N-DEPTH’s learned robustness to lossy compression expands to video codecs as well. Compared to an existing state-of-the-art encoding method, N-DEPTH achieves smaller file sizes and lower errors across a large range of compression qualities, in both image (JPEG) and video (H.264) formats. For example, reconstructions from N-DEPTH encodings stored with JPEG had dramatically lower error while still offering 29.8%-smaller file sizes. When H.264 video was used to target a 10 Mbps bit rate, N-DEPTH reconstructions had 85.1%-lower root mean square error (RMSE) and 15.3%-lower mean absolute error (MAE). Overall, our method offers an efficient and robust solution for emerging 3D streaming and 3D telepresence applications, enabling high-quality 3D depth data storage and transmission.

Deep Learning–Based Autonomous Road Condition Assessment Leveraging Inexpensive RGB and Depth Sensors and Heterogeneous Data Fusion: Pothole Detection and Quantification

Poor condition of roads is a major factor for traffic accidents and damage to vehicles. A significant portion of car accidents is attributed to severe three-dimensional (3D) pavement distresses such as potholes, ruttings, and ravelings. Insufficient road condition assessment is responsible for the poor condition of roads. To inspect the condition of the pavement surfaces more frequently and efficiently, an inexpensive data acquisition system was developed that consists of a consumer-grade RGB-D sensor and an edge computing device that can be mounted on vehicles and collect data while driving vehicles. The RGB-D sensor is used for collecting two-dimensional (2D) color images and corresponding 3D depth data, and the lightweight edge computing device is used to control the RGB-D sensor and store the collected data. An RGB-D pavement surface data set is generated. Furthermore, encoder-decoder deep convolutional neural networks (DCNNs) consisting of one or two encoders, and one decoder trained on heterogeneous RGB-D pavement surface data are used for pothole segmentation. Comprehensive experiments using different depth encoding techniques and data fusion methods including data- and feature-level fusion were performed to investigate the efficacy of defect detection using DCNNs. Experimental results demonstrate that the feature-level RGB-D data fusion based on the surface normal encoding of depth data outperform other approaches in terms of segmentation accuracy, where the mean intersection over union (IoU) over 10-fold cross-validation of 0.82 is achieved that shows a 7.7% improvement compared with a network trained only on RGB data. In addition, this study explores the efficacy of indirectly using depth information for pothole detection when depth data are not available. Additionally, the semantic segmentation results were utilized to quantify the severity level of the potholes assisting in maintenance decision-making. The result from these comprehensive experiments using an RGB-D pavement surface data set gathered through the proposed data acquisition system is a stepping stone for opportunistic data collection and processing through crowdsourcing and Internet of Things in future smart cities for effective road assessment. Finally, suggestions about the improvement of the proposed system are discussed.

가중 메디안 필터를 이용한 16채널 라이다의 깊이정보 업샘플링

Three-dimensional (3D) LiDAR is widely used because it can accurately measure 360° 3D depth. One of the most widely used sensors to this end is VLP-16, a 16-channel LiDAR manufactured by Velodyne. This 16-channel LiDAR provides dense 3D depth data in the horizontal direction. However, in the vertical direction, it provides only sparse depth data owing to the LiDAR architecture. We propose an upsampling method to obtain a dense 3D depth map from a 16-channel LiDAR by applying an edge-aware filter to the LiDAR data. A camera- LiDAR sensor system composed of a 16-channel LiDAR and six vision cameras is used for this purpose. First, the LiDAR depth data are projected onto the camera image coordinates to generate a 2.5D depth map. Next, bilinear interpolation and a weighted median filter are applied to upsample the original depth map. Compared with the original LiDAR data, we can generate an extremely dense depth map by using the proposed method. The results of quantitative analysis show that the proposed method can dramatically increase the number of 3D depth points and reduce the depth reconstruction error.

Adaptive Vision-Based Method for Rotor Dynamic Balance System

This article presents a new adaptive vision-based method (AVBM) of performing automatic detection for rotor balancing, and the online implementation proved that the method achieved rapid real-time optimization of system balancing configuration for a rotor dynamic balance machine. The proposed AVBM integrated 3D sensors and dynamic balancing platform using 3D computer vision technique and dynamic balance algorithm to improve the efficiency of rotor dynamic balancing. AVBM applied 3D ToF sensors on active rotor dynamic balance machines to grab 3D point cloud of rotor shaft and balance sprues. By 3D depth data, the background noise can be removed to detect the positions of shaft center, key phasor and balance sprues of rotor automatically. After combining with unbalance vector from dynamic balancing machine, the AVBM system calculated the optimal balance configuration by the vector analysis algorithm. Compares to conventional methods, conventional rotor dynamic balancing process relies on technicians to mount washers on particular balance sprues based on their experience, therefore uncertainty causes productivity decline. Experiments in industrial examples showed that the proposed AVBM required fewer rounds to achieve acceptance, whereas the conventional industrial rotor balancing method performed by operators required more than three rounds in average. Consider the overall dynamic balancing process for the motor, the processing time required for each motor without AVBM was 348.9 seconds, and the daily rotor balancing count of each dynamic balancing machine was 83. The processing time with AVBM was shortened from 348.9 to 283.9 seconds, the daily rotor balancing count had increased from 83 to 101, and the production improvement had reached 22 %. That is, the proposed AVBM can accurately analyze the dynamic balance and greatly reduce the redundant dynamic balance operations. The main advantage of the proposed AVBM over conventional methods is its efficiency, effectiveness and robustness in online optimization of rotor dynamic balance.

The CarbonNet Pelican 3D seismic survey – A first for Australia’s offshore greenhouse gas storage

The CarbonNet Pelican 3D seismic survey – A first for Australia’s offshore greenhouse gas storage

Techniques of structured light measurement network with 3D sensors

For achieving automatic, high-precision, high-density, and photorealistic 3D imaging and measurement of large-scale complex objects, the techniques about multi-node 3D sensor measurement network were described based on 3D sensing with the fringe structured light. It mainly involved the analysis of two key technologies (fringe analysis and phase reconstruction, system calibration and 3D reconstruction) of single 3D sensing, construction and optimization of multi-node 3D sensor measurement network, calibration of multi-node 3D sensor measurement network, matching and fusion of 3D depth data and texture data. Some experimental prototypes and experimental results were given.

Orientation Correction of Kinect’s 3D Depth Data for Mapping

Simultaneous localization and mapping (SLAM) is considered to be one of the primary tasks for autonomous robot navigation process. The first element required by the robot to start the navigation process is to create and assemble an accurate map of the surroundings or ground plane to envision its location. Mobile robots that use 1D or 2D sensors such as laser range finder, ultrasonic or sonar for the mapping process cannot provide enough information on the obstacle locations while 3D sensors like Kinect can. However, if the 3D sensor is not mounted correctly, the orientation of the 3D image obtained will also be affected and thus producing inaccurate maps. In this project, Microsoft Kinect was used to scan the environment. RANSAC algorithm was implemented to detect the ground plane. The orientation of the ground plane was corrected, and the depth data acquired by Kinect was then converted into 2D map. It was found that the methods applied have successfully mapped the obstacles detected.

딥러닝과 ICP 알고리즘을 이용한 링 모양의 소형 빈피킹 물체의 실시간 3차원 자세 추정

Bin picking is an important task in smart manufacturing and intelligent robotics. For a robot to pick or grip an object with a human-like gripping action, it needs to know the accurate 3D pose of the object. In this paper, we propose a method for estimating the 3D pose of a small ring-shaped object using infrared and depth images generated by a depth camera. The proposed method consists of two algorithm modules, the first to recognize an object in a 2D infrared image and the second to estimate the 3D pose by applying the ICP (iterative closest point) algorithm to 3D depth data. In the first module, we propose a method to generate a three-channel integrated image with features from the depth and infrared images. Next, we introduce a method for training an object detector based on deep-learning. Because the bin-picking test object in this paper is small and ring-shaped, it is difficult to detect and find 3D poses of individual objects when many such objects are piled up. We solved this problem with a depth-based filtering method. Using the filtered image, each object region is separated by the deep learning approach. In the second module, the ICP algorithm is employed to estimate the 3D pose of the ring object. We match a 3D reference model of the object and the real object using the point-to-point ICP algorithm. Performance of the proposed method is evaluated by using two different types of depth camera in the experiments.

Absolute Pose Estimation of Central Cameras Using Planar Regions.

A novel method is proposed for the absolute pose estimation of a central 2D camera with respect to 3D depth data without the use of any dedicated calibration pattern or explicit point correspondences. The proposed method has no specific assumption about the data source: plain depth information is expected from the 3D sensing device and a central camera is used to capture the 2D images. Both the perspective and omnidirectional central cameras are handled within a single generic camera model. Pose estimation is formulated as a 2D-3D nonlinear shape registration task which is solved without point correspondences or complex similarity metrics. It relies on a set of corresponding planar regions, and the pose parameters are obtained by solving an overdetermined system of nonlinear equations. The efficiency and robustness of the proposed method were confirmed on both large scale synthetic data and on real data acquired from various types of sensors.

3D gravity anomaly separation method taking into account the gravity response of the inhomogeneous mantle

In order to ensure that the gravity data processing and inversion are consistent with the actual geology, we propose that, in integrated geophysical research, 3D gravity anomaly separation (GAS) should take into account the gravity response of the inhomogeneous mantle. This is an effective method by which the residual gravity anomaly (RGA) may be obtained, as well as an important basis for the inversion and interpretation of gravity data. Aimed at the petroliferous basin structure, evolution of continental margin, ocean continental transition zone and other widely discussed geological problems, we extensively investigate the upper mantle structure in many regions around the world, and the results show that the lateral inhomogeneity of the mantle is widespread. In many comprehensive studies based on gravity and seismic data, the gravity anomaly modeling and separation are typically performed within the crust (above the Moho), which actually relies on an assumption of homogeneous mantle. Although this assumption simplifies the difficulty of the gravity modeling and separation, it ignores gravity anomalies caused by inhomogeneities of the deep mantle. The RGA, along with the inverted density structure and other interpretations, involves the risk of deviating from the actual geology. On the basis of the 2D forward calculation of several typical interface models, we implement the quantitative analysis of the effect that the inhomogeneous mantle has on the Bouguer gravity anomaly. In case studies of the southeastern United States and the Sichuan Basin, taking the shear wave velocity (Vs) structure of the upper mantle as the constraint condition, we calculate the gravity anomaly caused by the inhomogeneous mantle by means of the forward method. It is found that the gravity response of the inhomogeneous mantle cannot be neglected. Therefore, we conduct 2.5D joint simulation combining gravity and seismic data in the Sichuan Basin. The fitting results after separating the gravity response of the inhomogeneous mantle are clearly superior to those of the homogeneous mantle hypothesis. On the basis of the 3D depth data of several sedimentary strata, the Moho and the mantle high-velocity bodies, we implement 3D GAS, obtain the RGA of the basement in the Sichuan Basin, and discuss the geological origin of the “Dazu high gravity”.

INTEGRATING DEPTH AND IMAGE SEQUENCES FOR PLANETARY ROVER MAPPING USING RGB-D SENSOR

Abstract. RGB-D camera allows the capture of depth and color information at high data rates, and this makes it possible and beneficial integrate depth and image sequences for planetary rover mapping. The proposed mapping method consists of three steps. First, the strict projection relationship among 3D space, depth data and visual texture data is established based on the imaging principle of RGB-D camera, then, an extended bundle adjustment (BA) based SLAM method with integrated 2D and 3D measurements is applied to the image network for high-precision pose estimation. Next, as the interior and exterior elements of RGB images sequence are available, dense matching is completed with the CMPMVS tool. Finally, according to the registration parameters after ICP, the 3D scene from RGB images can be registered to the 3D scene from depth images well, and the fused point cloud can be obtained. Experiment was performed in an outdoor field to simulate the lunar surface. The experimental results demonstrated the feasibility of the proposed method.

Real-time Recognition and Pursuit in Robots Based on 3D Depth Data

In this work, we address the problem of robot pursuit based on a real-time object recognition system with 3D depth sensors. Compared with traditional RGBD data based recognition approaches, we propose a novel global online descriptor designed for object recognition from solely depth data. Proposed descriptor, which we name as Differential Histogram of Normal Vectors (DHONV), is designed to extract the geometric characteristics of the captured 3D surfaces of the objects presented in depth images. In order to obtain a brief description of the visible 3D surfaces of each object, we quantize the differential angles of the surface’s normal vectors into a 1D histogram. The object recognition experiments on a self-collected dataset and a benchmark RGB-D object dataset show that our proposed descriptor outperforms other depth data based descriptors. Moreover, we conducted real-time experiments with RoboCars. Our experiments with RoboCars validate our proposed method capability to perform a real-time recognition and pursuit tasks within indoor environment based solely on depth data.

An innovative Primary Surface Profile-based three-dimensional pavement distress data filtering approach for optical instruments and tilted pavement model-related noise reduction

The automatic pavement management system has the advantage of providing reliable pavement maintenance and rehabilitation strategies aiming at prolonging existing pavement service life. Therefore, the quality of noise reduction results, which is an unavoidable process of automatic pavement assessment evaluation, has a significant influence on the reliability of pavement maintenance operations suggested. The primary purpose of this paper is to propose an innovative three-dimensional (3D) pavement image-based data filtering protocol, thereby maintaining a highly functional pavement surface. First, a 3D pavement depth data collection system was developed using laser light and a charge-coupled device camera. After that, based on the analysis of Positive Noise and Negative Noise, which are optical instrument-related noises, and tilted pavement model noise, the Primary Surface Profile (PSP)-based raw data filtering approach was proposed which aims at improving the noise reduction quality. Validation experiments were conducted using both the proposed approach and the traditional data filtering method, and the results show that for the not tilted pavement surface model, the PSP-based filter method can achieve the highest noise reduction value (NRV), whereas for the tilted pavement surface model, with a slightly lower NRV than that of biphasic standard deviation average filtering, which demonstrates that the proposed data filtering method has self-adaptive and robust data filter advantages which can be incorporated into a high-performance pavement performance evaluation and management system.

点字ブロックを用いた移動ロボットの自律走行

It is difficult to know the self-position of the robot in a crowded environment. We use braille blocks as one way to navigate to the destination. The robot detects braille blocks using 3D sensor depth data. We consider them as a graph structure and autonomously navigate to the destination. There are common processing and induction block detector, warning block detector in the braille block detection processing. We use the Hough conversion in induction block detector, and clustering in warning block detector. As a result of the detection experiment, the detection rate was about 70%. We confirmed that we can navigate the robot to the destination by using the detector.

Mining Key Skeleton Poses with Latent SVM for Action Recognition

Human action recognition based on 3D skeleton has become an active research field in recent years with the recently developed commodity depth sensors. Most published methods analyze an entire 3D depth data, construct mid-level part representations, or use trajectory descriptor of spatial-temporal interest point for recognizing human activities. Unlike previous work, a novel and simple action representation is proposed in this paper which models the action as a sequence of inconsecutive and discriminative skeleton poses, named as key skeleton poses. The pairwise relative positions of skeleton joints are used as feature of the skeleton poses which are mined with the aid of the latent support vector machine (latent SVM). The advantage of our method is resisting against intraclass variation such as noise and large nonlinear temporal deformation of human action. We evaluate the proposed approach on three benchmark action datasets captured by Kinect devices: MSR Action 3D dataset, UTKinect Action dataset, and Florence 3D Action dataset. The detailed experimental results demonstrate that the proposed approach achieves superior performance to the state-of-the-art skeleton-based action recognition methods.

Using Visual Feature and Geometric Constraints for Robot Localization

This paper presents a novel method to generate an index word for the topological map in robot localization. Previous studies extract only appearance features from an input image to match the visual words of the model images. However, the localization performance is much affected by the miss or false matches. First, we segment a robot navigation environment into the structural planes using 3D depth data. We obtain both the surface normal vectors of the structural planes and visual features in the model image, which are compared with those of an input request image in the voting approach. The experimental results show the voting performance is improved by taking into account the spatial distribution of the features.

Fusing Multiple Features for Depth-Based Action Recognition

Human action recognition is a very active research topic in computer vision and pattern recognition. Recently, it has shown a great potential for human action recognition using the three-dimensional (3D) depth data captured by the emerging RGB-D sensors. Several features and/or algorithms have been proposed for depth-based action recognition. A question is raised: Can we find some complementary features and combine them to improve the accuracy significantly for depth-based action recognition? To address the question and have a better understanding of the problem, we study the fusion of different features for depth-based action recognition. Although data fusion has shown great success in other areas, it has not been well studied yet on 3D action recognition. Some issues need to be addressed, for example, whether the fusion is helpful or not for depth-based action recognition, and how to do the fusion properly. In this article, we study different fusion schemes comprehensively, using diverse features for action characterization in depth videos. Two different levels of fusion schemes are investigated, that is, feature level and decision level. Various methods are explored at each fusion level. Four different features are considered to characterize the depth action patterns from different aspects. The experiments are conducted on four challenging depth action databases, in order to evaluate and find the best fusion methods generally. Our experimental results show that the four different features investigated in the article can complement each other, and appropriate fusion methods can improve the recognition accuracies significantly over each individual feature. More importantly, our fusion-based action recognition outperforms the state-of-the-art approaches on these challenging databases.

Interactive lens through smartphones for supporting level-of-detailed views in a public display

AbstractIn this paper, we propose a new approach to providing interactive and collaborative lens among multi-users for supporting level-of-detailed views using smartphones in a public display. In order to provide smartphone-based lens capability, the locations of smartphones are effectively detected and tracked using Kinect, which provides RGB data and depth data (RGB-D). In particular, human skeleton information is extracted from the Kinect 3D depth data to calculate the smartphone location more efficiently and correctly with respect to the public display and to support head tracking for easy target selection and adaptive view generation. The suggested interactive and collaborative lens using smartphones not only can explore local spaces of the shared display but also can provide various kinds of activities such as LOD viewing and collaborative interaction. Implementation results are given to show the advantage and effectiveness of the proposed approach.

TriViews: A general framework to use 3D depth data effectively for action recognition

We present an effective framework to utilize 3D depth data for action recognition, called TriViews. It projects the 3D depth maps into three views, i.e., front, side, and top views. Under this framework, features can be extracted from each view, separately. Then the three views are combined to derive a complete description of the 3D data. To study the effectiveness and usefulness of the TriViews framework, we extract five different features, i.e., spatiotemporal interest points (STIP), dense trajectory shape (DT-Shape), dense trajectory motion boundary histograms (DT-MBH), skeleton trajectory shape (ST-Shape), and skeleton trajectory motion boundary histograms (ST-MBH). The first three features are representative for actions in intensity data but adapted to depth sequences. The last two are proposed by us, termed as skeleton-based features unique for 3D depth data. The RGB-D sensors, e.g., the Kinect, provide 3D positions of 20 skeleton joints and the evolution of each skeleton joint over time corresponds to one skeleton trajectory. Features aligned with the skeleton trajectory include shape descriptor (ST-Shape) and motion boundary histograms (ST-MBH), extracted to characterize the actions with sparse trajectories. The five features characterize action patterns from different aspects, among which the top three best features are selected and fused based on a probabilistic fusion approach (PFA). We evaluate the proposed framework on three challenging depth action datasets. The experimental results show that the proposed TriViews framework achieves the most accurate results for depth-based action recognition, better than the state-of-the-art methods on all three databases.

Strategy for dynamic 3D depth data matching towards robust action retrieval

3D depth data, especially dynamic 3D depth data, offer several advantages over traditional intensity videos for expressing objects׳ actions, such as being useful in low light levels, resolving the silhouette ambiguity of actions, and being color and texture invariant. With the wide popularity of somatosensory equipment (Kinect for example), more and more dynamic 3D depth data are shared on the Internet, which results in an urgent need to retrieve these data efficiently and effectively. In this paper, we propose a generalized strategy for dynamic 3D depth data matching and apply this strategy in action retrieval task. Firstly, an improved 3D shape context descriptor (3DSCD) is proposed to extract features of each static depth frame. Then we employ dynamic time warping (DTW) to measure the temporal similarity between two 3D dynamic depth sequences. Experimental results on our collected dataset consisting of 170 dynamic 3D depth video clips show that the proposed 3DSCD has a rich descriptive power on depth data and that the method using 3DSCD and DTW achieves high matching accuracy. Finally, to address the matching efficiency problem, we utilize the bag of word (BoW) model to quantize the 3DSCD of each static depth frame into visual word packages. So the original feature matching problem is simplified into a two-histogram matching problem. The results demonstrate the matching efficiency of our proposed method, while still maintaining high matching accuracy.