Image Depth Information Research Articles

Depth Image Assisted Aiming for Scoring Goal in Wheeled Soccer Robot

Wheeled soccer robot, as an automatic robot, required to have an advanced decision making system based on information it grasp from its surrounding. One of the most crucial decision making ability is to determine aiming angle when it is scoring goal.This research will enhance the aiming ability for scoring goal by predicting unguarded area of goal. Combination of depth image and RGB image information will be used to predict the position of unguarded space in goal. This position will be converted into aiming angle for the robot. Intel Realsense D435i depth camera will be used to get RGB and depth image simultaneouslyBy using this method, the system is capable to predict unguarded area in all of 60 test points, with 1.3% average error for the predicted coordinate.

Depth-Enhanced Deep Learning Approach For Monocular Camera Based 3D Object Detection

Automatic 3D object detection using monocular cameras presents significant challenges in the context of autonomous driving. Precise labeling of 3D object scales requires accurate spatial information, which is difficult to obtain from a single image due to the inherent lack of depth information in monocular images, compared to LiDAR data. In this paper, we propose a novel approach to address this issue by enhancing deep neural networks with depth information for monocular 3D object detection. The proposed method comprises three key components: 1)Feature Enhancement Pyramid Module: We extend the conventional Feature Pyramid Networks (FPN) by introducing a feature enhancement pyramid network. This module fuses feature maps from the original pyramid and captures contextual correlations across multiple scales. To increase the connectivity between low-level and high-level features, additional pathways are incorporated. 2)Auxiliary Dense Depth Estimator: We introduce an auxiliary dense depth estimator that generates dense depth maps to enhance the spatial perception capabilities of the deep network model without adding computational burden. 3)Augmented Center Depth Regression: To aid center depth estimation, we employ additional bounding box vertex depth regression based on geometry. Our experimental results demonstrate the superiority of the proposed technique over existing competitive methods reported in the literature. The approach showcases remarkable performance improvements in monocular 3D object detection, making it a promising solution for autonomous driving applications.

Finite-Time Synchronization of Neural Networks With Proportional Delays for RGB-D Image Protection.

Since the depth information of images facilitates the analysis of the spatial distance of objects in computer vision applications, it is necessary to protect the image depth information. Thus this article proposes a novel red-green-blue-depth (RGB-D) image protection algorithm, which is implemented with the finite-time synchronization (FTS) of neural networks (NNs) with proportional delays via the quantized intermittent control to derive the system synchronization criterion based on Lyapunov stability theory. The performance of RGB-D image protection depends on the synchronization error of the system by driving the system sequence to encrypt the RGB-D image and responding to the system sequence to decrypt the encrypted image. Subsequently, the validity of the proposed criteria is verified by simulation examples, and the practical application of RGB-D image protection is verified.

Hologram generation using light-field modulated zone plates

A hologram calculation technique is proposed in this study using a depth image and light-field information comprising multiple parallax images. The depth image is used to determine the phase distribution of the zone plates, and the amplitude distribution is modulated using the light-field information. An object wave of three-dimensional (3D) objects is obtained by adding the light-field modulated zone plates. The proposed technique can generate sharp and shaded 3D images in space. The occlusion effects of 3D images can also be supported by modifying the amplitude modulation scheme. The proposed technique was experimentally verified.

Toward Intraoperative Visual Intelligence: Real-Time Surgical Instrument Segmentation for Enhanced Surgical Monitoring.

Open surgery relies heavily on the surgeon's visual acuity and spatial awareness to track instruments within a dynamic and often cluttered surgical field. This system utilizes a head-mounted depth camera to monitor surgical scenes, providing both image data and depth information. The video captured from this camera is scaled down, compressed using MPEG, and transmitted to a high-performance workstation via the RTSP (Real-Time Streaming Protocol), a reliable protocol designed for real-time media transmission. To segment surgical instruments, we utilize the enhanced U-Net with GridMask (EUGNet) for its proven effectiveness in surgical tool segmentation. For rigorous validation, the system's performance reliability and accuracy are evaluated using prerecorded RGB-D surgical videos. This work demonstrates the potential of this system to improve situational awareness, surgical efficiency, and generate data-driven insights within the operating room. In a simulated surgical environment, the system achieves a high accuracy of 85.5% in identifying and segmenting surgical instruments. Furthermore, the wireless video transmission proves reliable with a latency of 200 ms, suitable for real-time processing. These findings represent a promising step towards the development of assistive technologies with the potential to significantly enhance surgical practice.

Glaucoma surgery experiments using digital microscope-integrated optical coherence tomography and OCT-compatible instruments

There is a certain failure rate in traditional glaucoma surgery because of the lack of depth information in microscope images. In this work, we present a digital microscope-integrated optical coherence tomography (MIOCT) system and several custom-made OCT-compatible instruments for glaucoma surgery. Sixteen ophthalmologists were asked to perform trabeculectomy and canaloplasty on live porcine eyes using the system and instruments. After surgery, a subjective feedback survey about the user experience was taken. The experiment results showed that our system can help surgeons easily locate important tissue structures during surgery. The custom-made instruments also solved the shadowing problem in OCT imaging. Surgeons preferred to use the system in their future practice.

MDP-SLAM: A Visual SLAM towards a Dynamic Indoor Scene Based on Adaptive Mask Dilation and Dynamic Probability

Visual simultaneous localization and mapping (SLAM) algorithms in dynamic scenes will apply the moving feature points to the camera pose’s calculation, which will cause the continuous accumulation of errors. As a target-detection tool, mask R-CNN, which is often used in combination with the former, due to the limited training datasets, easily results in the semantic mask being incomplete and deformed, which will increase the error. In order to solve the above problems, we propose in this paper a visual SLAM algorithm based on an adaptive mask dilation strategy and the dynamic probability of the feature points, named MDP-SLAM. Firstly, we use the mask R-CNN target-detection algorithm to obtain the initial mask of the dynamic target. On this basis, an adaptive mask-dilation algorithm is used to obtain a mask that can completely cover the dynamic target and part of the surrounding scene. Then, we use the K-means clustering algorithm to segment the depth image information in the mask coverage area into absolute dynamic regions and relative dynamic regions. Combined with the epipolar constraint and the semantic constraint, the dynamic probability of the feature points is calculated, and then, the highly dynamic possible feature points are removed to solve an accurate final pose of the camera. Finally, the method is tested on the TUM RGB-D dataset. The results show that the MDP-SLAM algorithm proposed in this paper can effectively improve the accuracy of attitude estimation and has high accuracy and robustness in dynamic indoor scenes.

FFLDGA-Net: Image retrieval method based on Feature Fusion Learnable Descriptor Graph Attention Network

Image retrieval aims to retrieve and return the image in the database that is most similar to the query image. However, the performance of image retrieval models is often hindered by the limited dimensionality of images, which lacks depth information about objects. To address this issue, we propose a novel image retrieval model called FFLDGA-Net (Feature Fusion-based Learnable Descriptor Graph Attention Network). This model aims to overcome the absence of depth information in images by fusing feature information from both image data and point cloud data. First, we introduce the LDGA-Net, which effectively improved the model’s ability to mine hard samples and negative samples. Then, we combine the multi-scale route convolution module with a one-dimensional path aggregation network to fuse point clouds and image features at multiple scales, and establish a high-dimensional relationship between features and low-dimensional features. To mitigate training noise, we incorporate a soft label strategy tailored to the dataset’s characteristics. Our experimental results on two benchmark datasets demonstrate the significant improvements achieved by FFLDGA-Net in image retrieval performance.

Research on YOLOv5 Vehicle Detection and Positioning System Based on Binocular Vision

Vehicle detection and location is one of the key sensing tasks of automatic driving systems. Traditional detection methods are easily affected by illumination, occlusion and scale changes in complex scenes, which limits the accuracy and robustness of detection. In order to solve these problems, this paper proposes a vehicle detection and location method for YOLOv5(You Only Look Once version 5) based on binocular vision. Binocular vision uses two cameras to obtain images from different angles at the same time. By calculating the difference between the two images, more accurate depth information can be obtained. The YOLOv5 algorithm is improved by adding the CBAM attention mechanism and replacing the loss function to improve target detection. Combining these two techniques can achieve accurate detection and localization of vehicles in 3D space. The method utilizes the depth information of binocular images and the improved YOLOv5 target detection algorithm to achieve accurate detection and localization of vehicles in front. Experimental results show that the method has high accuracy and robustness for vehicle detection and localization tasks.

AFC-ResNet18: A Novel Real-Time Image Semantic Segmentation Network for Orchard Scene Understanding

Highlights A novel real-time image semantic segmentation network for orchards, termed AFC-ResNet18, was designed and tested. The AFC-ResNet18 model outperformed the SwiftNet network in terms of segmentation depth. The AFC-ResNet18 model achieved the highest accuracy in the architecture performance testing. The AFC-ResNet18 model won first place in the orchard scene test with 72.5% accuracy. Abstract. Semantic segmentation is a fundamental prerequisite for the real-time understanding of scenes. This understanding is essential for developing automated devices that can enhance productivity. Orchards, being labor-intensive and time-consuming workplaces, urgently require automated equipment to boost efficiency. Therefore, the objective of this article is to develop a real-time image semantic segmentation network tailored for orchard environments. This development aims to offer significant new insights into the design of automated maintenance and harvesting equipment. Based on ResNet, the 2015 classification champion network, a novel real-time image semantic segmentation network termed AFC-ResNet18, which used an attentional feature complementary module (AFC) to fuse RGB and depth image information, was designed and systematically tested. Interestingly, in the segmentation ability tests, the AFC-ResNet18 model outperformed the SwiftNet network in terms of segmentation depth. Surprisingly, in the architecture performance testing, the AFC-ResNet18 model achieved the highest accuracy. Noteworthily, in the orchard scene test, the AFC-ResNet18 model won first place with 72.5% accuracy. Predictably, these findings may accelerate the development of novel automated equipment to maintain the orchard worldwide, especially AFC-ResNet18 based robots. Keywords: Attentional feature complementary module, Orchard, Real-time, Robots, Semantic segmentation.

Pose estimation via structure-depth information from monocular endoscopy images sequence.

Image-based endoscopy pose estimation has been shown to significantly improve the visualization and accuracy of minimally invasive surgery (MIS). This paper proposes a method for pose estimation based on structure-depth information from a monocular endoscopy image sequence. Firstly, the initial frame location is constrained using the image structure difference (ISD) network. Secondly, endoscopy image depth information is used to estimate the pose of sequence frames. Finally, adaptive boundary constraints are used to optimize continuous frame endoscopy pose estimation, resulting in more accurate intraoperative endoscopy pose estimation. Evaluations were conducted on publicly available datasets, with the pose estimation error in bronchoscopy and colonoscopy datasets reaching 1.43 mm and 3.64 mm, respectively. These results meet the real-time requirements of various scenarios, demonstrating the capability of this method to generate reliable pose estimation results for endoscopy images and its meaningful applications in clinical practice. This method enables accurate localization of endoscopy images during surgery, assisting physicians in performing safer and more effective procedures.

HAFE: A Hierarchical Awareness and Feature Enhancement Network for Scene Text Recognition

Scene text recognition (STR) is to obtain text information from natural text images. Compared to the optical character recognition, STR has attracted much attentions due to the variability of scene images. However, most STR methods tend to loss some spatial information due to only simply adding Resnet-45 to the front end of the encoder. To solve this problem, we propose a Hierarchical Awareness and Feature Enhancement (HAFE) network. Our proposed Hierarchical Awareness (HA) module can effectively extract the depth information of scene text images from multiple dimensions and then improve the anti-interference ability of the network. Besides, the proposed Feature Enhancement (FE) module can enhance the stability of the model through strengthening the global features, which is helpful to improve the accuracy of text recognition. On the whole, our HAFE network tends to strengthen the information in encoder, better learn the feature maps from different levels, and then improve the recognition accuracy of non-horizontal texts. Extensive comparative experiments on seven benchmark datasets show that the proposed method has the most advanced performance and outperforms most state-of-the-art ones. Our code will be released at website: https://github.com/HAFE in the future.

Achieving fine-grained urban flood perception and spatio-temporal evolution analysis based on social media

Timely understanding of affected areas during disasters is essential for the implementation of emergency response activities. As one of the low-cost and information-rich volunteer geographic information, social media data can reflect geographic events through human behavior, which is a powerful supplementary source for fine-grained flood monitoring in urban areas. However, the value of social media data has not been fully exploited as potential location and water depth information may be embedded in both text and images. In this study, we propose a novel framework for fine-grained information extraction and dynamic spatial-temporal awareness in disaster-stricken areas based on Sina Weibo. First, we construct a novel fine-grained location corpus specifically for urban flooding contexts. The corpus summarizes characteristics of address descriptions in flood-related Weibo texts, including standard address entities and spatial relationship entities, based on the named entity recognition (NER) model. Then, water depth information in texts and images is obtained based on different deep learning modules and fused at the decision level. Specifically, in text analysis module, we summarize and extract diverse descriptions of water depth, and in image analysis module, we develop a water level hierarchical mapping method. Finally, we analyze the spatio-temporal distribution characteristics and variation patterns of the extracted information to enhance situational awareness. Taking the urban flood occurred in Anhui, China as a case study, we find that the variation of flooding hotspot areas in Sina Weibo and rainfall centers show a significant spatial and temporal consistency, and the fusion of text and image-based information can facilitate dynamic perception of flood processes. The framework presented in this study provides a feasible way to implement refined situational awareness and spatio-temporal evolution analysis of urban floods at the city level in time.

EVALUATING MONOCULAR DEPTH ESTIMATION METHODS

Abstract. Depth estimation from monocular images has become a prominent focus in photogrammetry and computer vision research. Monocular Depth Estimation (MDE), which involves determining depth from a single RGB image, offers numerous advantages, including applications in simultaneous localization and mapping (SLAM), scene comprehension, 3D modeling, robotics, and autonomous driving. Depth information retrieval becomes especially crucial in situations where other sources like stereo images, optical flow, or point clouds are not available. In contrast to traditional stereo or multi-view methods, MDE techniques require fewer computational resources and smaller datasets. This research work presents a comprehensive analysis and evaluation of some state-of-the-art MDE methods, considering their ability to infer depth information in terrestrial images. The evaluation includes quantitative assessments using ground truth data, including 3D analyses and inference time.

DMNet: Dynamic Memory Network for RGB-D Salient Object Detection

RGB-D salient object detection aims to identify the most salient object using the color and depth information of images. Currently, most of the existing salient object detection models use the classical U-Net architecture, which uses up-sampling and short link decoding to exploit saliency cues after generating multi-level features by successive convolution and pooling operations. However, the multi-level encoding and progressive up-sampling decoding used by these models may lose some of the high-level semantic information during feature extraction and information fusion and some of the detailed information during image recovery, which in turn affects the quality of generated saliency maps. To solve these problems, we propose a dynamic memory network (DMNet) consisting of an interactive enhancement encoder and a dynamic memory decoder, which achieves high-precision detection of salient objects with a more complete and superior encoding and decoding strategy. Specifically, in the interactive enhancement encoder, we propose a feature interactive fusion module (FIFM), which can enhance intermediate-scale RGB features by fusing RGB features of three adjacent scales. Moreover, we design a depth-guided dense fusion module (DDFM) to fuse the features from RGB and depth modalities. In the dynamic memory decoder, we design a full-dimensional dynamic convolutional expansion module (FDCEM) to enhance the information representation capability of features at different scales. Furthermore, we also design a gated decoding module (GDM) to decode features of different scales and eliminate non-salient information. Extensive experimental results on STERE, SIP, NLPR, NJU2K, SSD, and DES datasets demonstrate that our model outperforms most of the state-of-the-art RGB-D SOD methods. In addition, the experimental results on three RGB-T datasets, VT821, VT1000, and VT5000 also show that our model can be effectively used for RGB-T SOD.

Buckwheat Plant Height Estimation Based on Stereo Vision and a Regression Convolutional Neural Network under Field Conditions

Buckwheat plant height is an important indicator for producers. Due to the decline in agricultural labor, the automatic and real-time acquisition of crop growth information will become a prominent issue for farms in the future. To address this problem, we focused on stereo vision and a regression convolutional neural network (CNN) in order to estimate buckwheat plant height. MobileNet V3 Small, NasNet Mobile, RegNet Y002, EfficientNet V2 B0, MobileNet V3 Large, NasNet Large, RegNet Y008, and EfficientNet V2 L were modified into regression CNNs. Through a five-fold cross-validation of the modeling data, the modified RegNet Y008 was selected as the optimal estimation model. Based on the depth and contour information of buckwheat depth image, the mean absolute error (MAE), root mean square error (RMSE), mean square error (MSE), and mean relative error (MRE) when estimating plant height were 0.56 cm, 0.73 cm, 0.54 cm, and 1.7%, respectively. The coefficient of determination (R2) value between the estimated and measured results was 0.9994. Combined with the LabVIEW software development platform, this method can estimate buckwheat accurately, quickly, and automatically. This work contributes to the automatic management of farms.

Cross-Modal Transformer for RGB-D semantic segmentation of production workshop objects

Scene understanding in a production workshop is an important technology to improve its intelligence level, semantic segmentation of production workshop objects is an effective method for realizing scene understanding. Since the varieties of information of production workshop, making full use of the complementary information of RGB image and depth image can effectively improve the semantic segmentation accuracy of production workshop objects. Aiming at solving the multi-scale and real-time problems of segmenting the production workshop objects, this paper proposes Cross-Modal Transformer (CMFormer), a Transformer-based cross-modal semantic segmentation model. Its key feature correction and feature fusion parts are composed of the Multi-Scale Channel Attention Correction(MS-CAC) module and the Global Feature Aggregation(GFA) module. By improving Multi-Head Self-Attention(MHSA) in Transformer, we design Cross-Modal Multi-Head Self-Attention(CM-MHSA) to build long-range interaction between RGB image and depth image, and further design the MS-CAC module and the GFA module on the basis of the CM-MHSA module, to achieve cross-modal information interaction in the channel and spatial dimensions. Among them, the MS-CAC module enriches the multi-scale features of each channel and achieve more accurate channel attention correction between the two modals; the GFA module interacts with RGB feature and depth feature in the spatial dimension and fuses global and local features at the same time. In the experiments on the NYU Depth v2 dataset, the CMFormer reached 68.00% MPA(Mean Pixel Accuracy) and 55.75% mIoU(Mean Intersection over Union), achieves the state-of-the-art results. While in the experiments on the Scene Objects for Production workshop dataset(SOP), the CMFormer achieves 96.74% MPA, 92.98% mIoU and 43 FPS(Frames Per Second), which has high precision and good real-time performance. Code is available at: https://github.com/FutureIAI/CMFormer

A Method for Obtaining 3D Point Cloud Data by Combining 2D Image Segmentation and Depth Information of Pigs.

This paper proposes a method for automatic pig detection and segmentation using RGB-D data for precision livestock farming. The proposed method combines the enhanced YOLOv5s model with the Res2Net bottleneck structure, resulting in improved fine-grained feature extraction and ultimately enhancing the precision of pig detection and segmentation in 2D images. Additionally, the method facilitates the acquisition of 3D point cloud data of pigs in a simpler and more efficient way by using the pig mask obtained in 2D detection and segmentation and combining it with depth information. To evaluate the effectiveness of the proposed method, two datasets were constructed. The first dataset consists of 5400 images captured in various pig pens under diverse lighting conditions, while the second dataset was obtained from the UK. The experimental results demonstrated that the improved YOLOv5s_Res2Net achieved a mAP@0.5:0.95 of 89.6% and 84.8% for both pig detection and segmentation tasks on our dataset, while achieving a mAP@0.5:0.95 of 93.4% and 89.4% on the Edinburgh pig behaviour dataset. This approach provides valuable insights for improving pig management, conducting welfare assessments, and estimating weight accurately.

In-plane Strain Analysis by Correlating Geometry and Visual Data Through a Gradient-Based Surface Reconstruction.

Abnormalities in tissue can be detected and analyzed by evaluating mechanical properties, such as strain and stiffness. While current sensor systems are effective in measuring longitudinal properties perpendicular to the measurement sensor, identifying in-plane deformation remains a significant challenge. To address this issue, this paper presents a novel method for reconstructing in-plane deformation of observed tissue surfaces using a fringe projection sensor specifically designed for measuring tissue deformations. The method employs the latest techniques from computer vision, such as differentiable rendering, to formulate the in-plane reconstruction as a differentiable optimization problem. This enables the use of gradient-based solvers for an efficient and effective optimization of the problem optimum. Depth information and image information are combined using landmark correspondences between the respective image observations of the undeformed and deformed scenes. By comparing the reconstructed pre- and post-deformation geometry, the in-plane deformation can be revealed through the analysis of relative variations between the corresponding models' geometries. The proposed reconstruction pipeline is validated on an experimental setup, and the potential for intraoperative applications is discussed.

Depth and Radius Joint Estimation for Underground Pipeline Using GPR

In this paper, a deep learning-based methodology is proposed to estimate depth and radius for underground pipeline using Ground penetrating radar(GPR) images. The proposed methodology conducts the depth and radius joint estimation based on GPR B-scan image preprocessing. During the preprocessing, the autoencoder structure network is designed to binarize the GPR image. Besides, redundant detection boxes of hyperbola detection are also eliminated. For pipeline depth estimation, the parabola model is used instead of the hyperbola model to obtain the symmetry axis of the pipeline GRP image more efficiently. The consecutive a-scan data at extracted symmetry axis and its vicinity are spliced and sent to the Long Short-Term Memory Network for depth estimation. For pipeline radius estimation, the image features and depth information are concatenated. A pre-trained convolutional neural network is designed to extract the GPR image features. Meanwhile, a 2d-Mapping network is proposed to encode the 1-dimensional depth information into 2-dimensional feature matrix. Improving the dimension of depth information rather than flattening the GPR image features (dimension reduction) can preserve the spatial information of image features. Besides, the depth information can be concatenated with the image features more flexibly. In all, the proposed methodology can accurately estimate the depth and radius of underground pipeline. To verify the effectiveness of the proposed methodology, the gprMax is used to generate the pipeline GPR images in inhomogeneous soil and create the dataset. The experiments include detailed comparison and analysis, illustrating that the proposed methodology is feasible and effective.