Depth Images Research Articles

NeRF-OR: neural radiance fields for operating room scene reconstruction from sparse-view RGB-D videos.

RGB-D cameras in the operating room (OR) provide synchronized views of complex surgical scenes. Assimilation of this multi-view data into a unified representation allows for downstream tasks such as object detection and tracking, pose estimation, and action recognition. Neural radiance fields (NeRFs) can provide continuous representations of complex scenes with limited memory footprint. However, existing NeRF methods perform poorly in real-world OR settings, where a small set of cameras capture the room from entirely different vantage points. In this work, we propose NeRF-OR, a method for 3D reconstruction of dynamic surgical scenes in the OR. Where other methods for sparse-view datasets use either time-of-flight sensor depth or dense depth estimated from color images, NeRF-OR uses a combination of both. The depth estimations mitigate the missing values that occur in sensor depth images due to reflective materials and object boundaries. We propose to supervise with surface normals calculated from the estimated depths, because these are largely scale invariant. We fit NeRF-OR to static surgical scenes in the 4D-OR dataset and show that its representations are geometrically accurate, where state of the art collapses to sub-optimal solutions. Compared to earlier work, NeRF-OR grasps fine scene details while training 30 faster. Additionally, NeRF-OR can capture whole-surgery videos while synthesizing views at intermediate time values with an average PSNR of 24.86dB. Last, we find that our approach has merit in sparse-view settings beyond those in the OR, by benchmarking on the NVS-RGBD dataset that contains as few as three training views. NeRF-OR synthesizes images with a PSNR of 26.72dB, a 1.7% improvement over state of the art. Our results show that NeRF-OR allows for novel view synthesis with videos captured by a small number of cameras with entirely different vantage points, which is the typical camera setting in the OR. Code is available via: github.com/Beerend/NeRF-OR .

Research on the Depth Image Reconstruction Algorithm Using the Two-Dimensional Kaniadakis Entropy Threshold.

The photon-counting light laser detection and ranging (LiDAR), especially the Geiger mode avalanche photon diode (Gm-APD) LiDAR, can obtain three-dimensional images of the scene, with the characteristics of single-photon sensitivity, but the background noise limits the imaging quality of the laser radar. In order to solve this problem, a depth image estimation method based on a two-dimensional (2D) Kaniadakis entropy thresholding method is proposed which transforms a weak signal extraction problem into a denoising problem for point cloud data. The characteristics of signal peak aggregation in the data and the spatio-temporal correlation features between target image elements in the point cloud-intensity data are exploited. Through adequate simulations and outdoor target-imaging experiments under different signal-to-background ratios (SBRs), the effectiveness of the method under low signal-to-background ratio conditions is demonstrated. When the SBR is 0.025, the proposed method reaches a target recovery rate of 91.7%, which is better than the existing typical methods, such as the Peak-picking method, Cross-Correlation method, and the sparse Poisson intensity reconstruction algorithm (SPIRAL), which achieve a target recovery rate of 15.7%, 7.0%, and 18.4%, respectively. Additionally, comparing with the SPIRAL, the reconstruction recovery ratio is improved by 73.3%. The proposed method greatly improves the integrity of the target under high-background-noise environments and finally provides a basis for feature extraction and target recognition.

Deep learning-based sow posture classifier using colour and depth images

Assessing sow posture is essential for understanding their physiological condition and helping farmers improve herd productivity. Deep learning-based techniques have proven effective for image interpretation, offering a better alternative to traditional image processing methods. However, distinguishing transitional postures such as sitting and kneeling is challenging with only conventional top-view RGB images. This study aimed to develop and compare different deep learning-based sow posture classifiers using different architectures and image types. Using Kinect v.2 cameras, RGB and depth images were collected from 9 sows housed individually in farrowing crates. A total of 26,362 images were manually labelled by posture: “standing”, “kneeling”, “sitting”, “ventral recumbency” and “lateral recumbency”. Different deep learning algorithms were developed to detect sow postures from three types of images: colour (RGB), depth (depth image transformed into greyscale), and fused (colour-depth composite images). Results indicated that the ResNet-18 model presented the best results and that including depth information improved the performance of all models tested. Depth and fused models achieved higher accuracies than the models using only RGB images. The best model used only depth images as input and presented an accuracy of 98.3 %. The mean precision and recall values were 97.04 % and 97.32 %, respectively (F1-score = 97.2 %). The study shows improved posture classification using depth images. Future research can improve model accuracy and speed by expanding the database, exploring fused methods and computational models, considering different breeds of sows, and incorporating more postures. These models can be integrated into computer vision systems to automatically characterise sow behavior.

Self-adaptive 2D[sbnd]3D image fusion for automated pixel-level pavement crack detection

Current 2D and 3D image-based crack detection methods in transportation infrastructure often struggle with noise robustness and feature diversity. To overcome these challenges, the paper use CSF-CrackNet, a self-adaptive 2D3D image fusion model utilizes channel and spatial modules for automated pavement crack segmentation. CSF-CrackNet consists of four parts: feature enhanced and field sensing (FEFS) module, channel module, spatial module, and semantic segmentation module. A multi-feature image dataset was established using a vehicle-mounted 3D imaging system, including color images, depth images, and color-depth overlapped images. Results show that the mean intersection over union (mIOU) of most models under the CSF-CrackNet framework can be increased to above 80 %. Compared with original RGB and depth images, the average mIOU increases with image fusion by 10 % and 5 %, respectively. The ablation experiment and weight significance analysis further demonstrate that CSF-CrackNet can significantly improve semantic segmentation performance by balancing information between 2D and 3D images.

Evaluation of lamina cribrosa parameters, nerve fiber thickness, and macular thickness in primary open-angle glaucoma and pseudoexfoliation glaucoma using optical coherence tomography.

To evaluate the lamina cribrosa, retinal nerve fiber layer (RNFL), and macula in patients with primary open-angle glaucoma (POAG) and pseudoexfoliation glaucoma (PEXG) and healthy individuals using enhanced depth imaging (EDI) of spectral-domain optical coherence tomography (SD-OCT). A total of 158 eyes were included in the study, comprising 58 eyes of 29 patients with POAG, 50 eyes of 25 patients with PEXG, and 50 eyes of 25 healthy individuals. The lamina cribrosa thickness (LCT) (at three locations), lamina cribrosa depth (LCD), RNFL thickness, and the macular thickness were measured using the EDI mode of the SD-OCT. The results were compared among the three groups. In both POAG and PEXG groups, the LCT was significantly thinner in the center, mid-superior, and mid-inferior areas in both eyes than in the control group (p < 0.001). However, no statistically significant difference was observed between the POAG and PEXG groups in terms of LCT at all three measurement locations in both eyes (p > 0.05). The LCD was significantly lower in the control group compared to the POAG and PEXG groups (p < 0.05), but there was no significant difference between the POAG and PEXG groups (p > 0.05). The RNFL thickness was significantly lower in both the POAG and PEXG groups compared to the control group in both eyes (p < 0.05). The LCT and LCD of patients with POAG and PEXG were thinner than those of healthy individuals, but there was no significant difference between the patients with POAG and PEXG.

Advancing in RGB-D Salient Object Detection: A Survey

The human visual system can rapidly focus on prominent objects in complex scenes, significantly enhancing information processing efficiency. Salient object detection (SOD) mimics this biological ability, aiming to identify and segment the most prominent regions or objects in images or videos. This reduces the amount of data needed to process while enhancing the accuracy and efficiency of information extraction. In recent years, SOD has made significant progress in many areas such as deep learning, multi-modal fusion, and attention mechanisms. Additionally, it has expanded in real-time detection, weakly supervised learning, and cross-domain applications. Depth images can provide three-dimensional structural information of a scene, aiding in a more accurate understanding of object shapes and distances. In SOD tasks, depth images enhance detection accuracy and robustness by providing additional geometric information. This additional information is particularly crucial in complex scenes and occlusion situations. This survey reviews the substantial advancements in the field of RGB-Depth SOD, with a focus on the critical roles played by attention mechanisms and cross-modal fusion methods. It summarizes the existing literature, provides a brief overview of mainstream datasets and evaluation metrics, and quantitatively compares the discussed models.

Sequential Two-Mode Fusion Underwater Single-Photon Lidar Imaging Algorithm

Aiming at the demand for long-range and high-resolution imaging detection of small targets such as submerged submarine markers in shallow coastal waters, research on single-photon lidar imaging technology is carried out. This paper reports the sequential two-mode fusion imaging algorithm, which has a strong information extraction capability and can reconstruct scene target depth and reflection intensity images from complex signal photon counts. The algorithm consists of four steps: data preprocessing, extremely large group value estimation, noise sieving, and total variation smoothing constraints to image the target with high quality. Simulation and test results show that the imaging performance and imaging characteristics of the method are better than the current high-performance first-photon group imaging algorithm, indicating that the method has a great advantage in sparse photon counting imaging, and the method proposed in this paper constructs a clear depth and reflectance intensity image of the target scene, even in the 50,828 Lux ambient strong light and strong interference, the 0.1 Lux low-light environment, or the underwater high-attenuation environment.

Enhancing Facial Expression Recognition through Light Field Cameras.

In this paper, we study facial expression recognition (FER) using three modalities obtained from a light field camera: sub-aperture (SA), depth map, and all-in-focus (AiF) images. Our objective is to construct a more comprehensive and effective FER system by investigating multimodal fusion strategies. For this purpose, we employ EfficientNetV2-S, pre-trained on AffectNet, as our primary convolutional neural network. This model, combined with a BiGRU, is used to process SA images. We evaluate various fusion techniques at both decision and feature levels to assess their effectiveness in enhancing FER accuracy. Our findings show that the model using SA images surpasses state-of-the-art performance, achieving 88.13% ± 7.42% accuracy under the subject-specific evaluation protocol and 91.88% ± 3.25% under the subject-independent evaluation protocol. These results highlight our model's potential in enhancing FER accuracy and robustness, outperforming existing methods. Furthermore, our multimodal fusion approach, integrating SA, AiF, and depth images, demonstrates substantial improvements over unimodal models. The decision-level fusion strategy, particularly using average weights, proved most effective, achieving 90.13% ± 4.95% accuracy under the subject-specific evaluation protocol and 93.33% ± 4.92% under the subject-independent evaluation protocol. This approach leverages the complementary strengths of each modality, resulting in a more comprehensive and accurate FER system.

Reconstruction of natural images from human fMRI using a three-stage multi-level deep fusion model

BackgroundImage reconstruction is a critical task in brain decoding research, primarily utilizing functional magnetic resonance imaging (fMRI) data. However, due to challenges such as limited samples in fMRI data, the quality of reconstruction results often remains poor. New methodWe proposed a three-stage multi-level deep fusion model (TS-ML-DFM). The model employed a three-stage training process, encompassing components such as image encoders, generators, discriminators, and fMRI encoders. In this method, we incorporated distinct supplementary features derived separately from depth images and original images. Additionally, the method integrated several components, including a random shift module, dual attention module, and multi-level feature fusion module. ResultsIn both qualitative and quantitative comparisons on the Horikawa17 and VanGerven10 datasets, our method exhibited excellent performance.Comparison with existing methods: For example, on the primary Horikawa17 dataset, our method was compared with other leading methods based on metrics the average hash value, histogram similarity, mutual information, structural similarity accuracy, AlexNet(2), AlexNet(5), and pairwise human perceptual similarity accuracy. Compared to the second-ranked results in each metric, the proposed method achieved improvements of 0.99 %, 3.62 %, 3.73 %, 2.45 %, 3.51 %, 0.62 %, and 1.03 %, respectively. In terms of the SwAV top-level semantic metric, a substantial improvement of 10.53 % was achieved compared to the second-ranked result in the pixel-level reconstruction methods. ConclusionsThe TS-ML-DFM method proposed in this study, when applied to decoding brain visual patterns using fMRI data, has outperformed previous algorithms, thereby facilitating further advancements in research within this field.

Pachychoroid pigment epitheliopathy in keratoconic eyes.

To report the association between keratoconus (KC) and pachychoroid pigment epitheliopathy (PPE). As secondary outcome, we explored the relation between subfoveal choroidal thickness (SFCT) and topometric indexes. Retrospective, observational, cross-sectional, case-control study. Multicentric study including patients with KC and healthy controls. Each subject underwent a complete ophthalmological visit, Placido-based corneal topography, Scheimpflug corneal tomography and spectral-domain OCT (SD-OCT) with the enhanced depth imaging (EDI) mode on. Linear mixed models (LMM) were employed for comparison between groups, and to examine the impact of different topometric factors on SFCT. KC stages were defined according to Belin grading. Overall, 56 eyes from 35 KC patients and 52 eyes from 27 healthy, age- and axial length-matched control subjects were included in the study. PPE was found in 10 (17.9%) eyes from 8 KC patients, whereas was absent in all healthy controls. SFCT was statistically significantly higher in keratoconic eyes (median: 390µm; interquartile range (IQR): 339 - 425µm) compared to healthy eyes (median: 240µm; IQR: 200 - 288µm) (p < 0.001). SFCT did not differ across different KC stages and between keratoconic eyes with and without PPE. PPE is a relatively frequent finding in eyes with KC, being present in about 1 out of 6 cases, regardless of disease stage. What is known: Keratoconus (KC) is a corneal disorder commonly associated with other chorioretinal abnormalities. It is well known that keratoconic eyes display increased choroidal thickness, and the rare association between KC and central serous chorioretinopathy (CSC) has already been reported in the literature. We identified an association between KC and pachychoroid pigment epitheliopathy (PPE), a precursor or forme fruste of pachychoroid diseases, such as CSC. PPE presence is independent from corneal parameters and is observed in about 1 out 6 KC eyes.

Integration of Stacked Object Recognition and Robotic Arm Gripping System Utilizing Mask Region-based Convolutional Neural Network and Red–Green–Blue Depth Images

Integration of Stacked Object Recognition and Robotic Arm Gripping System Utilizing Mask Region-based Convolutional Neural Network and Red–Green–Blue Depth Images

Ocular Involvement in Patients with Infantile Nephropathic Cystinosis.

Cystinosis is a rare autosomal recessive lysosomal storage disease associated with high mortality and morbidity rates. The most distinctive ocular manifestations of cystinosis are photophobia, tearing, and blurred vision. Herein, we assessed the ocular involvement of four patients from two families diagnosed with infantile nephropathic cystinosis using optical coherence tomography (OCT) and in vivo confocal microscopy (IVCM). Anterior segment OCT demonstrated multiple hyperreflective punctate deposits, and IVCM revealed needle-shaped bright crystal deposits in the corneal stroma in all patients. Three patients also had crystal deposits in the epithelium, where epithelial cell disruption was observed. Crystal deposits around the subepithelial nerve plexus were noted in some sections. In one patient, round and needle-shaped bright deposits along with inflammatory cells were observed in the limbal region of the conjunctiva. Infrared fundus images of two female siblings revealed hyperreflective crystal-like deposits around the optic disc, macula, and peripheral retina, and enhanced depth imaging OCT showed accumulation of crystals in all layers of the retina.

Developing Machine Vision in Tree-Fruit Applications-Fruit Count, Fruit Size and Branch Avoidance in Automated Harvesting.

Recent developments in affordable depth imaging hardware and the use of 2D Convolutional Neural Networks (CNN) in object detection and segmentation have accelerated the adoption of machine vision in a range of applications, with mainstream models often out-performing previous application-specific architectures. The need for the release of training and test datasets with any work reporting model development is emphasized to enable the re-evaluation of published work. An additional reporting need is the documentation of the performance of the re-training of a given model, quantifying the impact of stochastic processes in training. Three mango orchard applications were considered: the (i) fruit count, (ii) fruit size and (iii) branch avoidance in automated harvesting. All training and test datasets used in this work are available publicly. The mAP 'coefficient of variation' (Standard Deviation, SD, divided by mean of predictions using models of repeated trainings × 100) was approximately 0.2% for the fruit detection model and 1 and 2% for the fruit and branch segmentation models, respectively. A YOLOv8m model achieved a mAP50 of 99.3%, outperforming the previous benchmark, the purpose-designed 'MangoYOLO', for the application of the real-time detection of mango fruit on images of tree canopies using an edge computing device as a viable use case. YOLOv8 and v9 models outperformed the benchmark MaskR-CNN model in terms of their accuracy and inference time, achieving up to a 98.8% mAP50 on fruit predictions and 66.2% on branches in a leafy canopy. For fruit sizing, the accuracy of YOLOv8m-seg was like that achieved using Mask R-CNN, but the inference time was much shorter, again an enabler for the field adoption of this technology. A branch avoidance algorithm was proposed, where the implementation of this algorithm in real-time on an edge computing device was enabled by the short inference time of a YOLOv8-seg model for branches and fruit. This capability contributes to the development of automated fruit harvesting.

Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images.

Pose estimation is crucial for ensuring passenger safety and better user experiences in semi- and fully autonomous vehicles. Traditional methods relying on pose estimation from regular color images face significant challenges due to a lack of three-dimensional (3D) information and the sensitivity to occlusion and lighting conditions. Depth images, which are invariant to lighting issues and provide 3D information about the scene, offer a promising alternative. However, there is a lack of strong work in 3D pose estimation from such images due to the time-consuming process of annotating depth images with 3D postures. In this paper, we present a novel approach to 3D human posture estimation using depth and infrared (IR) images. Our method leverages a three-stage fine-tuning process involving simulation data, approximated data, and a limited set of manually annotated samples. This approach allows us to effectively train a model capable of accurate 3D pose estimation with a median error of under 10 cm across all joints, using fewer than 100 manually annotated samples. To the best of our knowledge, this is the first work focusing on vehicle occupant posture detection utilizing only depth and IR data. Our results demonstrate the feasibility and efficacy of this approach, paving the way for enhanced passenger safety in autonomous vehicle systems.

An accurate measurement method of irregular coal gangue volume based on adaptive image restoration and error compensation

To improve the accuracy of volume measurement for irregularly shaped coal gangue, defects in the depth image and various errors in volume measurement are analyzed. The gray difference similarity (Gds) is proposed for pre-classifying defects, and the depth and RGB information are used interactively for adaptive repair. To minimize the impact of abnormal gray distribution and irrelevant information on the repair process, the areas to be repaired are divided into abnormal regions or layers based on gray and distance weights. Taking into account parameters like geometric morphology (GeMo), volume (Ve), and GeMo × Ve, the influence of wrong imaging and gaps is fully considered. The reverse compensation of errors is achieved by applying the principles of mathematical statistics and the nonlinear surface fitting algorithm. The results show that compared with other algorithms, the proposed algorithm has the smallest measurement error of 6.614 % and is not easily affected by external factors.

Automated Measurement of Cattle Dimensions Using Improved Keypoint Detection Combined with Unilateral Depth Imaging.

Traditional measurement methods often rely on manual operations, which are not only inefficient but also cause stress to cattle, affecting animal welfare. Currently, non-contact cattle dimension measurement usually involves the use of multi-view images combined with point cloud or 3D reconstruction technologies, which are costly and less flexible in actual farming environments. To address this, this study proposes an automated cattle dimension measurement method based on an improved keypoint detection model combined with unilateral depth imaging. Firstly, YOLOv8-Pose is selected as the keypoint detection model and SimSPPF replaces the original SPPF to optimize spatial pyramid pooling, reducing computational complexity. The CARAFE architecture, which enhances upsampling content-aware capabilities, is introduced at the neck. The improved YOLOv8-pose achieves a mAP of 94.4%, a 2% increase over the baseline model. Then, cattle keypoints are captured on RGB images and mapped to depth images, where keypoints are optimized using conditional filtering on the depth image. Finally, cattle dimension parameters are calculated using the cattle keypoints combined with Euclidean distance, the Moving Least Squares (MLS) method, Radial Basis Functions (RBFs), and Cubic B-Spline Interpolation (CB-SI). The average relative errors for the body height, lumbar height, body length, and chest girth of the 23 measured beef cattle were 1.28%, 3.02%, 6.47%, and 4.43%, respectively. The results show that the method proposed in this study has high accuracy and can provide a new approach to non-contact beef cattle dimension measurement.

Online chicken carcass volume estimation using depth imaging and 3-D reconstruction

Variability in the size of slaughtered chickens remains a longstanding challenge in the standardization of the poultry industry. To address this issue, we present a novel approach that uses volume as a grading metric for chicken carcasses. This innovative method, unexplored in existing studies, employs real-time data capture of moving chicken carcasses on a production line using Kinect v2 depth imaging and 3-D reconstruction technologies. The captured depth images are processed into point clouds followed by 3-D reconstruction. Volume is calculated from the reconstructed models using the surface integration method, and additional 2-D and 3-D features are extracted as input parameters for machine learning models. Multiple regression models were evaluated, with the bagged tree model demonstrating superior performance, achieving an R² value of 0.9988, RMSE of 5.335, and ARE of 2.125%. Furthermore, our method showed remarkable efficiency with an average processing time of less than 1.6 seconds per carcass. These results indicate that our novel approach fills a critical gap in existing automated grading methodologies by offering both accuracy and efficiency. This validates the applicability of depth imaging, 3-D reconstruction, and machine learning for estimating chicken carcass volume with high precision, thereby enabling a more comprehensive, efficient, and reliable chicken carcass grading system.

Depth Image Rectification Based on an Effective RGB–Depth Boundary Inconsistency Model

Depth image has been widely involved in various tasks of 3D systems with the advancement of depth acquisition sensors in recent years. Depth images suffer from serious distortions near object boundaries due to the limitations of depth sensors or estimation methods. In this paper, a simple method is proposed to rectify the erroneous object boundaries of depth images with the guidance of reference RGB images. First, an RGB–Depth boundary inconsistency model is developed to measure whether collocated pixels in depth and RGB images belong to the same object. The model extracts the structures of RGB and depth images, respectively, by Gaussian functions. The inconsistency of two collocated pixels is then statistically determined inside large-sized local windows. In this way, pixels near object boundaries of depth images are identified to be erroneous when they are inconsistent with collocated ones in RGB images. Second, a depth image rectification method is proposed by embedding the model into a simple weighted mean filter (WMF). Experiment results on two datasets verify that the proposed method well improves the RMSE and SSIM of depth images by 2.556 and 0.028, respectively, compared with recent optimization-based and learning-based methods.

RGBD-based method for segmenting apparent pores within bridge towers

Abstract As a crucial technology in computer vision, image semantic segmentation is extensively applied to tasks such as detecting apparent defects in concrete, identifying structural cracks, and interpreting facility scenes within infrastructure settings. Challenges such as uneven lighting inside bridge towers and the similarity in color and texture between concrete pore structures and their surrounding areas frequently result in lower segmentation accuracy. This article introduces a multimodal semantic segmentation model incorporating depth information to tackle these challenges. By integrating depth and RGB images as inputs, the model constructs an interactive space and utilizes a cross-attention mechanism along with global context for guided learning, thus enabling precise feature extraction and segmentation. The experimental results demonstrate that this segmentation network excels on a custom-made concrete pore dataset, with a precision rate of 90.88%, recall rate of 87.48%, intersection over union (IoU) of 80.42%, and F1 and Dice coefficient of 89.10% and 89.15%, respectively. These outcomes affirm the network’s effectiveness in segmenting apparent defects in concrete and offer technical support and solutions for image semantic segmentation tasks in settings like bridge towers.

Depth estimation from monocular endoscopy using simulation and image transfer approach

Obtaining accurate distance or depth information in endoscopy is crucial for the effective utilization of navigation systems. However, due to space constraints, incorporating depth cameras into endoscopic systems is often impractical. Our goal is to estimate depth images directly from endoscopic images using deep learning. This study presents a three-step methodology for training a depth-estimation network model. Initially, simulated endoscopy images and corresponding depth maps are generated using Unity based on a colon surface model obtained from segmented computed tomography colonography data. Subsequently, a cycle generative adversarial network model is employed to enhance the realism of the simulated endoscopy images. Finally, a deep learning model is trained using the synthesized endoscopy images and depth maps to estimate depths accurately. The performance of the proposed approach is evaluated and compared against prior studies utilizing unsupervised training methods. The results demonstrate the superior precision of the proposed technique in estimating depth images within endoscopy. The proposed depth estimation method holds promise for advancing the field by enabling enhanced navigation, improved lesion marking capabilities, and ultimately leading to better clinical outcomes.