RGB-D Images Research Articles

An Improved High Precision 3D Semantic Mapping of Indoor Scenes from RGB-D Images

An Improved High Precision 3D Semantic Mapping of Indoor Scenes from RGB-D Images

FHENet: Lightweight Feature Hierarchical Exploration Network for Real-Time Rail Surface Defect Inspection in RGB-D Images

In recent years, computer vision systems have been increasingly applied to rail defect inspection. Rail defects should be identified quickly and accurately to ensure safe, stable, and fast train operations and thereby reduce the incidence of accidents and economic losses. As most existing methods focus on accuracy, they cannot be deployed on mobile devices with limited computational resources. To solve this problem, we propose a lightweight feature hierarchical exploration network (FHENet) for real-time rail surface defect inspection. First, boundary textures of rail defects are acquired based on the maximum function and maximum pooling in a novel boundary extraction module (BEM), which improves boundary prediction while avoiding heavy computations. Second, a novel cross-modality exploration module (CEM) complements prominent regions through basic operations to avoid complex inference while providing high detection performance. Third, a novel multifeature integration module (MIM) optimizes representative feature regions by using simple operations to avoid complex computations. Results from extensive experiments demonstrate the superiority of the proposed FHENet to 14 state-of-the-art methods. Regarding efficiency, FHENet outperforms the state-of-the-art methods with only 5.26 M parameters and a processing speed of 60.33 frames/s. The FHENet code and results are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hjklearn/Rail-Defect-Detection</uri> .

3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction

With the rapid development of virtual reality (VR) technology, VR headsets, a.k.a. Head-Mounted Displays (HMDs), are widely available, allowing immersive 3D content to be viewed. A natural need for truly immersive VR is to allow bidirectional communication: the user should be able to interact with the virtual world using facial expressions and eye gaze, in addition to traditional means of interaction. The typical application scenario includes VR virtual conferencing and virtual roaming, where ideally users are able to see other users expressions and have eye contact with them in the virtual world. In addition, eye gaze also provides a natural means of interaction with virtual objects. Despite significant achievements in recent years for reconstruction of 3D faces from RGB or RGB-D images, it remains a challenge to reliably capture and reconstruct 3D facial expressions including eye gaze when the user is wearing an HMD, because the majority of the face is occluded, especially those areas around the eyes which are essential for recognizing facial expressions and eye gaze. In this paper, we introduce a novel real-time system that is able to capture and reconstruct 3D faces wearing HMDs, and robustly recover eye gaze. We further propose a novel method to map eye gaze directions to the 3D virtual world, which provides a novel and useful interactive mode in VR. We compare our method with state of-the-art techniques both qualitatively and quantitatively, and demonstrate the effectiveness of our system using live capture.

Digital model reconstruction through 3D Stereo Depth camera: a faster method exploiting robot poses

In industrial robotic contact-based operations it is necessary to have a detailed geometrical knowledge of the work-piece to automate the generation of the working trajectory. In many cases, the digital model is not available or it differs from its as built actual conditions. Vision sensors, especially 3D vision sensors allows to scan the work-piece and reconstruct its digital copy that is used for the generation of the robotic working trajectory. In this paper we compare two algorithms for the generation of the 3D model of an unknown work-piece based on the use of the RGB-D images taken from different perspectives. The first technique, based on standard image reconstruction methods commonly used for reconstruction of indoor scenes where error in the order of few centimeters is negligible, is exploited in this work in order to be exploited in contact-based robotic operations where the geometrical error has to be limited to few millimeters. It is based on the analysis of the images to estimate the camera pose taking each image. Based on the pose, the algorithm integrates then the information contained in the images in a unique volume representing the scene. The second algorithm directly uses the known poses of the robot, stored while capturing each image, to integrate the information to create the model. The two algorithms are compared to evaluate the accuracy, acquisition time and elaboration time. The analysis is done considering two kinds of objects, the first one with regular shape, the second one corresponding to an actual industrial object.

Robust Symmetry Prediction with Multi-Modal Feature Fusion for Partial Shapes

In geometry processing, symmetry research benefits from global geometric features of complete shapes, but the shape of an object captured in real-world applications is often incomplete due to the limited sensor resolution, single viewpoint, and occlusion. Different from the existing works predicting symmetry from the complete shape, we propose a learning approach for symmetry prediction based on a single RGB-D image. Instead of directly predicting the symmetry from incomplete shapes, our method consists of two modules, i.e., the multi-modal feature fusion module and the detection-by-reconstruction module. Firstly, we build a channel-transformer network (CTN) to extract cross-fusion features from the RGB-D as the multi-modal feature fusion module, which helps us aggregate features from the color and the depth separately. Then, our self-reconstruction network based on a 3D variational auto-encoder (3D-VAE) takes the global geometric features as input, followed by a prediction symmetry network to detect the symmetry. Our experiments are conducted on three public datasets: ShapeNet, YCB, and ScanNet, we demonstrate that our method can produce reliable and accurate results.

A Novel Semantic Segmentation Algorithm for RGB-D Images Based on Non-Symmetry and Anti-Packing Pattern Representation Model

With the rapid development of deep learning technology, the accuracy of image semantic segmentation tasks has been greatly improved. However, indoor RGB-D semantic segmentation remains a challenging problem because of the complexity of indoor environments. The emergence of depth sensors makes depth information gradually used to improve the effect of semantic segmentation. The splicing of weights such as between the RGB features and the depth features which are used as the input features of the neural network can effectively improve the accuracy of the indoor semantic segmentation tasks. Most previous researches focused on improving the performance of semantic segmentation by adjusting the structure of convolutional neural network. These researches have either added attention mechanism or performed data augmentation on input features, but they didn’t make full use of the boundary information and the texture information of the original RGB image. In this paper, we propose a semantic segmentation algorithm for RGB-D images based on Non-symmetry and Anti-packing pattern representation Model (NAM). The core idea of the proposed algorithm is to take the channel-wise concatenation of pre-segmentation labels provided by the traditional hierarchical image segmentation and RGB-D features as the input of the neural network so as to guide the semantic segmentation tasks. The extensive experiments are conducted on the popular indoor RGB-D semantic segmentation datasets. When compared with the state-of-art algorithms, the experimental results presented in this paper show that our proposed method has improved the performance of image semantic segmentation networks on several popular neural network architectures.

Miner detection in underground coal mine by fusing RGBD image with improved convolutional block attention module

Miner detection in underground coal mine by fusing RGBD image with improved convolutional block attention module

Toward 3D scene reconstruction from locally scale-aligned monocular video depth

Monocular depth estimation methods have achieved excellent robustness on diverse scenes, usually by predicting affine-invariant depth, up to an unknown scale and shift, rather than metric depth in that it is much easier to collect large-scale affine-invariant depth training data. However, in some video-based scenarios such as video depth estimation and 3D scene reconstruction, the unknown scale and shift residing in per-frame prediction may cause the predicted depth to be inconsistent. To tackle this problem, we propose a locally weighted linear regression method to recover the scale and shift map with very sparse anchor points, which ensures the consistency along consecutive frames. Extensive experiments show that our method can drop the Rel error (relative error) of existing state-of-the-art approaches significantly over several zero-shot benchmarks. Besides, we merge 6.3 million RGBD images to train robust depth models. By locally recovering scale and shift, our produced ResNet50-backbone model even outperforms the state-of-the-art DPT ViT-Large model. Combined with geometry-based reconstruction methods, we formulate a new dense 3D scene reconstruction pipeline, which benefits from both the scale consistency of sparse points and the robustness of monocular methods. By performing simple per-frame prediction over a video, the accurate 3D scene geometry can be recovered.

3D Spine Model Reconstruction Based on RGBD Images of Unclothed Back Surface.

Most 3D spine reconstruction methods require X-ray images as input, which usually leads to high cost and radiation damage. Therefore, these methods are hardly ever applied to large scale scoliosis screening or spine pose monitoring during treatment. We propose a novel, low-cost, easy-to-operate and none-radioactive 3D spine model reconstruction method, which is based on human back surface information without requiring X-ray images as input. Our method fits a pre-built Spine Priors Model (SPrM) to human back surface information and reconstructs the main part of spine with 17 vertebrae: lumbar vertebrae L1-L5 and thoracic vertebrae T1-T12. The Spine Priors Model is constructed according to human spine priors, including Statistical Spine Shape Model (SSSM), Spine Pose Model (SPM) and Spine Biomechanical Simplified Model (SBSM). The spine-related information on back surface, including back surface spinous curve and local symmetry nearby spinous curve is extracted from the RGBD images of human back surface. We formulate the spine optimization constraints from spine-related feature on back surface and spine priors, then optimize the spine model by gradient descent to get the optimal personalized shape parameters and pose parameters of the Spine Priors Model (SPrM). We assess our reconstruction by scoliosis Cobb angle error, and the result is comparable to current X-ray based methods.

3D object detection through fog and occlusion: passive integral imaging vs active (LiDAR) sensing.

In this paper, we address the problem of object recognition in degraded environments including fog and partial occlusion. Both long wave infrared (LWIR) imaging systems and LiDAR (time of flight) imaging systems using Azure Kinect, which combine conventional visible and lidar sensing information, have been previously demonstrated for object recognition in ideal conditions. However, the object detection performance of Azure Kinect depth imaging systems may decrease significantly in adverse weather conditions such as fog, rain, and snow. The concentration of fog degrades the depth images of Azure Kinect camera, and the overall visibility of RGBD images (fused RGB and depth image), which can make object recognition tasks challenging. LWIR imaging may avoid these issues of lidar-based imaging systems. However, due to poor spatial resolution of LWIR cameras, thermal imaging provides limited textural information within a scene and hence may fail to provide adequate discriminatory information to identify between objects of similar texture, shape and size. To improve the object detection task in fog and occlusion, we use three-dimensional (3D) integral imaging (InIm) system with a visible range camera. 3D InIm provides depth information, mitigates the occlusion and fog in front of the object, and improves the object recognition capabilities. For object recognition, the YOLOv3 neural network is used for each of the tested imaging systems. Since the concentration of fog affects the images from different sensors (visible, LWIR, and Azure Kinect depth cameras) in different ways, we compared the performance of the network on these images in terms of average precision and average miss rate. For the experiments we conducted, the results indicate that in degraded environment 3D InIm using visible range cameras can provide better image reconstruction as compared to the LWIR camera and Azure Kinect RGBD camera, and therefore it may improve the detection accuracy of the network. To the best of our knowledge, this is the first report comparing the performance of object detection between passive integral imaging system vs active (LiDAR) sensing in degraded environments such as fog and partial occlusion.

Sparse RGB-D images create a real thing: A flexible voxel based 3D reconstruction pipeline for single object

Reconstructing 3D models for single objects with complex backgrounds has wide applications like 3D printing, AR/VR, and so on. It is necessary to consider the tradeoff between capturing data at low cost and getting high-quality reconstruction results. In this work, we propose a voxel-based modeling pipeline with sparse RGB-D images to effectively and efficiently reconstruct a single real object without the geometrical post-processing operation on background removal. First, referring to the idea of VisualHull, useless and inconsistent voxels of a targeted object are clipped. It helps focus on the target object and rectify the voxel projection information. Second, a modified TSDF calculation and voxel filling operations are proposed to alleviate the problem of depth missing in the depth images. They can improve TSDF value completeness for voxels on the surface of the object. After the mesh is generated by the MarchingCube, texture mapping is optimized with view selection, color optimization, and camera parameters fine-tuning. Experiments on Kinect capturing dataset, TUM public dataset, and virtual environment dataset validate the effectiveness and flexibility of our proposed pipeline.

RGB-D salient object ranking based on depth stack and truth stack for complex indoor scenes

RGB-D salient object detection has achieved a great development in recent years due to its extensive applications. Previous studies mainly focus on simple scene images with one single object. These models usually become overwhelmed by complex scenes with multiple objects. Moreover, these methods model salient object detection as a binary segmentation problem. However, psychology studies show that humans shift their visual attention from one object to another and rank salient objects, especially in complex indoor scenes. Following the psychological studies, we propose to rank salient objects in RGB-D images of complex indoor scenes. Due to the lack of such data, we first construct a RGB-D salient object ranking dataset containing complex indoor scenes with multiple objects. The saliency ranking of different objects is defined based on the order that an observer notices these objects. The final salient object ranking result is an average across the saliency rankings of 13 observers. This RGB-D salient object ranking dataset is also analyzed with current mainstream RGB-D salient object detection dataset for comparison. Since location information provided by depth images can help to determine the saliency ranking of objects, we further propose an end-to-end network exploiting depth stack and ground truth stack to predict the order of salient objects in complex scenes. The quantitative and qualitative comparisons demonstrate the effectiveness of the proposed method.

Anatomy aware-based 2.5D bronchoscope tracking for image-guided bronchoscopic navigation

ABSTRACT Physicians use an endoscopic navigation system during bronchoscopy to decrease the risk of getting lost in complex tree-structure like bronchus. Most existing navigation systems based on the camera pose estimated from bronchoscope tracking and/or deep learning. However, bronchoscope tracking-based method exists tracking error, and the pre-training of the model needs massive data. This paper describes an improved bronchoscope tracking procedure by adopting image domain translation technique to improve tracking performance. Specifically, our scheme consists of three modules, an RGB-D image domain translation module, an anatomical structure classification module and a structure-aware bronchoscope tracking module. The RGB-D image domain translation module translates a real bronchoscope (RB) image to its corresponding virtual bronchoscope image and depth image. The anatomical dependency module classifies the current scene into two categories: structureless and rich structure. The bronchoscope tracking module uses a modified video-CT bronchoscope tracking approach to estimate camera pose. Experimental results showed that the proposed method achieved higher tracking accuracy than the current state-of-the-art bronchoscope tracking methods.

CNN-based object detection and growth estimation of plum fruit (Prunus mume) using RGB and depth imaging techniques

Modern people who value healthy eating habits have shown increasing interest in plum (Prunus mume) fruits, primarily owing to their nutritiousness and proven efficacy. As consumption increases, it becomes important to monitor work to prevent Prunus mume fruits from falling out. Moreover, determining the growth status of Prunus mume is also crucial and is attracting increasing attention. In this study, convolutional neural network (CNN)-based deep learning object detection was developed using RGBD images collected from Prunus mume farms. These RGBD images consider various environments, including the depth information of objects in the outdoor field. A faster region-based convolutional neural network (R-CNN), EfficientDet, Retinanet, and Single Shot Multibox Detector (SSD) were applied for detection, and the performance of all models was estimated by comparing their respective computing speeds and average precisions (APs). The test results show that the EfficientDet model is the most accurate, and SSD MobileNet is the fastest among the four models. In addition, the algorithm was developed to acquire the growth status of P. mume fruits by applying the coordinates and score values of bounding boxes to the depth map. Compared to the diameters of the artificial Prunus mume fruits used as the experimental group, the calculated diameters were very similar to those of the artificial objects. Collectively, the results demonstrate that the CNN-based deep learning Prunus mume detection and growth estimation method can be applied to real farmlands.

Real-Time Robotic Multigrasp Detection Using Anchor-Free Fully Convolutional Grasp Detector

Robotic grasping is essential for intelligent manufacturing. This article presents a novel anchor-free grasp detector based on fully convolutional network for detecting multiple valid grasps from RGB-D images in real time. Grasp detection is formulated as a closest horizontal or vertical rectangle regression task and a grasp angle classification task. By directing predicting grasps at feature points, our method eliminates the predefined anchors that commonly used in prior methods, and thus anchor-related hyperparameters and complex computations are avoided. For suppressing ambiguous and low-quality training samples, a new sample assignment strategy that combines center-sampling and regression weights is proposed. Our method achieves a state-of-the-art accuracy of 99.4% on Cornell and 96.2% on Jacquard dataset, and real-time speed of 104 frames per second, with approximately 2× fewer parameters and 8× less training time compared to previous one-stage detector. Moreover, an efficient multiscale feature fusion module is integrated to improve the performance of multigrasp detection by 25%. In real-world robotic grasping of novel objects, our method achieves a grasp success rate of 91.3% for single object and 83.3% for multiple objects with only 26 ms used for the whole planning. The results demonstrate that our method is robust for potential industrial applications.

Automatic Depth Image Generation for Unseen Object Amodal Instance Segmentation

This research proposes an automatic instance-wise depth image estimation method for Unseen Object Amodal Instance Segmentation (UOAIS), which predicts depth images automatically by sharing features of other prediction branches. The depth images can provide valuable information for amodal mask generation. To achieve good performance, the model estimates the depth images in two steps by adding two extra depth branches to the original UOAIS model. A new feature fusion scheme incorporating these two branches is also designed to facilitate information sharing between tasks. The new model is evaluated on OCID and UOAIS-Sim evaluation datasets. Compared with the performance of the original model with pure RGB input, the AP50 of the new model increases by 0.685% on the UOAIS-Sim evaluation dataset, and the overlap and boundary F measures increase by 2.3% and 1.6% respectively on the OCID dataset. Consequently, the model achieves similar performance to those which need RGB-D images as input. It has the potential to serve as a compromise plan for UOAIS if the collection of real scene-depth images is difficult or unavailable.

Space Noncooperative Object Active Tracking With Deep Reinforcement Learning

Actively tracking an arbitrary space noncooperative object relied on visual sensor remains a challenging problem. In this article, we provide an open-source benchmark for space noncooperative object visual tracking including simulated environment, evaluation toolkit, and a position-based visual servoing (PBVS) baseline algorithm, which can facilitate the research in this topic especially for those methods based on deep reinforcement learning. We also present an end-to-end active visual tracker based on deep Q-learning, named as DRLAVT, which learns approximately optimal policy merely took color or RGBD images as input. To the best of authors knowledge, it is the first intelligent agent used for active visual tracking in aerospace domain. The experiment results show that our DRLAVT achieves an excellent robustness and real-time performance compared with the PBVS baseline, benefitted from the design of complex neural network and efficient reward function. In addition, the multiple targets training adopted in this article effectively guarantees the transferability of DRLAVT by forcing the agent to learn optimal control policy with respect to motion patterns of the target.

MRI and CT Image Based Breast Tumor Detection Framework with Boundary Detection Technique

Image segmentation is vital in image processing and computer vision, and it is also regarded as a bottleneck in image processing technology development. Picture segmentation is the process of dividing an image into a group of disjoint sections with uniform and homogeneous characteristics. Before proceeding with various statistical methods of analyzing segmentation of tumor, one has to understand the labels consisting of brain MR image. Because of the high inconstancy in tumor morphology and the low sign to-commotion proportion characteristic to mammography, manual characterization of mammogram yields a critical number of patients being gotten back to, and consequent enormous number of biopsies performed to decrease the danger of missing malignant growth. The convolutional neural networks (CNN) is a mainstream profound learning build utilized in picture arrangement. This procedure has accomplished huge progressions in enormous set picture arrangement challenges in later a long time. In this examination, we had acquired more than 3000 excellent unique mammograms with endorsement from an institutional survey board at the University of Kentucky. Various classifiers dependent on CNNs were manufactured, and every classifier was assessed dependent on its exhibition comparative with truth esteems created by histology results from biopsy furthermore, two-year negative mammogram follow-up affirmed by master. In this paper, a method for classifying traffic signs is proposed that is based on training the convolutional neural networks (CNN). Furthermore, it shows the preliminary classification performance of using this CNN to automatically learn and categorise RGB-D images. For this four-class classification job, the method of transfer learning known as fine tuning technique is proposed which involves reusing layers learnt on the ImageNet dataset to discover the optimal design.

Multi-scale fusion for RGB-D indoor semantic segmentation

In computer vision, convolution and pooling operations tend to lose high-frequency information, and the contour details will also disappear with the deepening of the network, especially in image semantic segmentation. For RGB-D image semantic segmentation, all the effective information of RGB and depth image can not be used effectively, while the form of wavelet transform can retain the low and high frequency information of the original image perfectly. In order to solve the information losing problems, we proposed an RGB-D indoor semantic segmentation network based on multi-scale fusion: designed a wavelet transform fusion module to retain contour details, a nonsubsampled contourlet transform to replace the pooling operation, and a multiple pyramid module to aggregate multi-scale information and context global information. The proposed method can retain the characteristics of multi-scale information with the help of wavelet transform, and make full use of the complementarity of high and low frequency information. As the depth of the convolutional neural network increases without losing the multi-frequency characteristics, the segmentation accuracy of image edge contour details is also improved. We evaluated our proposed efficient method on commonly used indoor datasets NYUv2 and SUNRGB-D, and the results showed that we achieved state-of-the-art performance and real-time inference.

Software Tool for the Extrinsic Calibration of Infrared and RGBD Cameras Applied to Thermographic Inspection

Context: Thermographic inspections are currently used to assess energy efficiency in electrical equipment and civil structures or to detect failures in cooling systems and electrical or electronic devices. However, thermal images lack texture details, which does not allow for a precise identification of the geometry of the scene or the objects in it. Method: In this work, the development of the software tool called DepTherm is described. This tool allows performing intrinsic and extrinsic calibration between infrared and RGBD cameras in order to fuse thermal, RGB, and RGBD images, as well as to record thermal and depth data. Additional features include user management, a visualization GUI for all three types of images, database storage, and report generation. Results: In addition to the integration tests performed to validate the functionality of DepTherm, two quantitative tests were conducted in order to evaluate its accuracy. A maximum re-projection error of 1,47±0,64 pixels was found, and the maximum mean error in registering an 11 cm side cube was 4,15 mm. Conclusions: The features of the DepTherm software tool are focused on facilitating thermographic inspections by capturing 3D scene models with thermal data.