Fusion Of Depth Images Research Articles

Height estimation of sugarcane tip cutting position based on multimodal alignment and depth image fusion

Sugarcane tip cutting is essential to reducing the rate of impurities in the harvest. To achieve adaptive regulation of the tip-cutting position by a sugarcane harvester, we propose a method for estimating the height of the tip-cutting position of sugarcane. The RGB and Binocular depth cameras are aligned to process the sugarcane tip region image. This involves threshold segmentation, morphological operations, and contour detection to identify the tip-cutting position and upper boundary contours. The depth image is segmented using contour pixel information and merged to form a colour depth image of the sugarcane's tip. This image is then transformed using depth data and triangular parallax principles to determine the height of the sugarcane tip-cutting position. The proposed method was evaluated in various sugarcane plantation environments. Comparative analysis between the proposed method and manual measurements of actual cutting position heights revealed that the RMSE ranged from 1.22 cm to 1.78 cm, and R2 varied between 0.79 and 0.86. These results demonstrate the effectiveness of the proposed method in accurately extracting the height information of sugarcane tip-cutting positions, which has a specific application value for the adaptive adjustment of the tip-cutting device of the sugarcane harvester.

Norest-Net: Normal Estimation Neural Network for 3-D Noisy Point Clouds.

The widely deployed ways to capture a set of unorganized points, e.g., merged laser scans, fusion of depth images, and structure-from- x , usually yield a 3-D noisy point cloud. Accurate normal estimation for the noisy point cloud makes a crucial contribution to the success of various applications. However, the existing normal estimation wisdoms strive to meet a conflicting goal of simultaneously performing normal filtering and preserving surface features, which inevitably leads to inaccurate estimation results. We propose a normal estimation neural network (Norest-Net), which regards normal filtering and feature preservation as two separate tasks, so that each one is specialized rather than traded off. For full noise removal, we present a normal filtering network (NF-Net) branch by learning from the noisy height map descriptor (HMD) of each point to the ground-truth (GT) point normal; for surface feature recovery, we construct a normal refinement network (NR-Net) branch by learning from the bilaterally defiltered point normal descriptor (B-DPND) to the GT point normal. Moreover, NR-Net is detachable to be incorporated into the existing normal estimation methods to boost their performances. Norest-Net shows clear improvements over the state of the arts in both feature preservation and noise robustness on synthetic and real-world captured point clouds.

Sparse-to-Dense Multi-Encoder Shape Completion of Unstructured Point Cloud

Unstructured point clouds are a representative shape representation of real-world scenes in 3D vision and graphics. Incompletion inevitably arises, due to the way the set of unorganized points is captured, e.g., as fusion of depth images, merged laser scans, or structure-from-x. In this paper, an end-to-end sparse-to-dense multi-encoder neural network (termed an SDME-Net) is proposed for uniformly completing an unstructured point cloud with its shape details preserved. Unlike most existing learning-based shape completion methods that are enforced on the representations of 2D images and 3D voxelization of point clouds, and require priors of the underlying shape's structures, topologies and annotations, the SDME-Net is implemented on the incomplete and even noisy point cloud without any transformation, and makes no specific assumptions about the incompletion distribution and geometry features in the input. Specifically, the defective point cloud is completed and optimized in a sparse-to-dense manner of two-stages. In the first stage, we generate a sparse but complete point cloud based on a bistratal PointNet, and in the second stage, we yield a dense and high-fidelity point cloud by encoding and decoding the sparse result in the first stage using PointNet++. Meanwhile, we combine the distance loss and repulsion loss to generate more uniformly distributed output point clouds closer to the ground-truth counterparts. Qualitative and quantitative experiments on the public ShapeNet dataset illustrate that our approach outperforms the state-of-art learning-based point cloud shape completion methods in terms of real structure recovery, uniformity, and noise/partiality robustness.

Bimanual gesture recognition based on convolution neural network

Gesture recognition is a key research field in the human-computer interaction. At present, most of researchers focus on one-handed gesture recognition, but do not pay much attention to bimanual (two hands) gesture recognition. This paper presents a deep learning-based solution to tackle the self-occlusion and self-similarity. To solve this problem, this paper uses Kinect to collect many colour and depth images of different gestures, and each gesture contains multiple sample individuals. Colour images and depth images are used to train the recognition model of bimanual gesture respectively, and then the colour image and depth image are fused, and the bimanual gesture recognition model is trained based on colour image and depth image fusion. Then, the bimanual recognition effects of the three models are compared. The experimental results show that, regardless of the single gesture precision or the mean average precision, the bimanual gesture recognition effect of the fused model is better than the gesture recognition models based on either colour image or depth image.

Action Recognition Using Weighted Fusion of Depth Images and Skeleton's Key Frames

This paper presents a new method for human action recognition fusing depth images and skeletal maps. Each depth image is represented by 2D and 3D auto-correlation of gradients features. A feature using spatial and orientational auto-correlation is extracted from depth images. Mutual information is used to define the similarity of each frame in the skeleton sequence, and then extract the key frames from the skeleton sequence. The skeleton feature extracted from the key frames as complementary features to cope with the temporal information loss in depth images. Each set of feature is used as input to two extreme learning machine classifiers and assign different weight to each set of feature. Using different classifier weights provides more flexible to different features. The final class label is determined according to the fused result. Experiments conducted on MSR_Action3D depth action data set show the accuracy of this proposed method is 1.5% higher than the state-of-the-art action recognition methods.