Scene Depth Research Articles

A robust approach for balancing the color and light integrity of underwater images

Abstract As light travels under the deep water, it scatters and is absorbed, resulting in a loss of intensity and altered color perception—a phenomenon known as underwater light attenuation. Images captured under these low light conditions suffered from color distortions, as you go deeper, colors fade in this order: red, orange, and yellow, while green and blue become more prominent. The red channel experiences significant attenuation due to the scattering properties of light under the deep water. As a consequence, deep water images often display noticeable color casts. Researchers encounter various challenges when enhancing low-light underwater images, such as reduced contrast, detail loss, artifacts, noise, and color distortion. In this paper, we present a novel Adaptive Color and Light Correction (ACLC) method for color correction and an Intuitionistic Fuzzy Generator (IFG) for enhancing low-light underwater images. The proposed Adaptive Color and Light Correction (ACLC) method tackles color casts on individual pixels by considering the scene depth. The proposed Intuitionistic Fuzzy Generator (IFG) method balances the image contrast by computing an intuitionistic fuzzy image representation using the proposed IFG approach. Experimental results reveal that the proposed approach significantly improves the color quality and contrast of the output image. The proposed ACLC and IFG methods exceed existing underwater color correction and low-light image enhancement techniques in visual and quantitative evaluations, as evidenced by extensive experimentation on well-established underwater image datasets, such as UIEB, Ocean dark, and LSUI.

Fluorescence microscopic image enhancement method based on multi-saliency guided filtering fusion

Abstract Aiming at addressing the issues of colour deviation and clarity reduction in digital pathological imaging systems caused by complex environmental interferences such as uneven slice thickness, sample jitter, and background noise, this paper proposes a multi-saliency-guided filtering fusion enhancement method that combines energy saliency and Gaussian saliency. Firstly, a series of Gamma correction and spatial linear saturation adjustments are applied to enhance the contrast and colour saturation of blurred images, resulting in a sequence of pseudo-exposure images with improved contrast and saturation. Then, the pseudo-exposure images are decomposed into base and detail layers, and the fusion rules of energy saliency and Gaussian saliency-guided filtering are applied to each layer separately. Finally, clear microscopic images are obtained through two-scale reconstruction. This method overcomes the limitations of conventional deblurring algorithms that rely on prior estimation of scene depth and complex depth mapping processes. Experimental results on a dataset of cytological bright-field microscopic imaging samples show that the clarity evaluation metrics, including peak signal-to-noise ratio, root mean square error, and structural similarity index, have significantly improved, indicating that the proposed method can effectively improve the colour fidelity of the samples while eliminating noise interference and enhancing the clarity of cellular texture structures.

FaSS-MVS: Fast Multi-View Stereo with Surface-Aware Semi-Global Matching from UAV-Borne Monocular Imagery.

With FaSS-MVS, we present a fast, surface-aware semi-global optimization approach for multi-view stereo that allows for rapid depth and normal map estimation from monocular aerial video data captured by unmanned aerial vehicles (UAVs). The data estimated by FaSS-MVS, in turn, facilitate online 3D mapping, meaning that a 3D map of the scene is immediately and incrementally generated as the image data are acquired or being received. FaSS-MVS is composed of a hierarchical processing scheme in which depth and normal data, as well as corresponding confidence scores, are estimated in a coarse-to-fine manner, allowing efficient processing of large scene depths, such as those inherent in oblique images acquired by UAVs flying at low altitudes. The actual depth estimation uses a plane-sweep algorithm for dense multi-image matching to produce depth hypotheses from which the actual depth map is extracted by means of a surface-aware semi-global optimization, reducing the fronto-parallel bias of Semi-Global Matching (SGM). Given the estimated depth map, the pixel-wise surface normal information is then computed by reprojecting the depth map into a point cloud and computing the normal vectors within a confined local neighborhood. In a thorough quantitative and ablative study, we show that the accuracy of the 3D information computed by FaSS-MVS is close to that of state-of-the-art offline multi-view stereo approaches, with the error not even an order of magnitude higher than that of COLMAP. At the same time, however, the average runtime of FaSS-MVS for estimating a single depth and normal map is less than 14% of that of COLMAP, allowing us to perform online and incremental processing of full HD images at 1-2 Hz.

Sequential Two-Mode Fusion Underwater Single-Photon Lidar Imaging Algorithm

Aiming at the demand for long-range and high-resolution imaging detection of small targets such as submerged submarine markers in shallow coastal waters, research on single-photon lidar imaging technology is carried out. This paper reports the sequential two-mode fusion imaging algorithm, which has a strong information extraction capability and can reconstruct scene target depth and reflection intensity images from complex signal photon counts. The algorithm consists of four steps: data preprocessing, extremely large group value estimation, noise sieving, and total variation smoothing constraints to image the target with high quality. Simulation and test results show that the imaging performance and imaging characteristics of the method are better than the current high-performance first-photon group imaging algorithm, indicating that the method has a great advantage in sparse photon counting imaging, and the method proposed in this paper constructs a clear depth and reflectance intensity image of the target scene, even in the 50,828 Lux ambient strong light and strong interference, the 0.1 Lux low-light environment, or the underwater high-attenuation environment.

PSD-ELGAN: A pseudo self-distillation based CycleGAN with enhanced local adversarial interaction for single image dehazing

Compared to pixel-level content loss, domain-level style loss in CycleGAN-based dehazing algorithms just imposes relatively soft constraints on the intermediate translated images, resulting in struggling to accurately model haze-free features from real hazy scenes. Furthermore, globally perceptual discriminator may misclassify real hazy images with significant scene depth variations as clean style, thereby resulting in severe haze residue. To address these issues, we propose a pseudo self-distillation based CycleGAN with enhanced local adversarial interaction for image dehazing, termed as PSD-ELGAN. On the one hand, we leverage the characteristic of CycleGAN to generate pseudo image pairs during training. Knowledge distillation is employed in this unsupervised framework to transfer the informative high-quality features from the self-reconstruction network of real clean images to the dehazing generator of paired pseudo hazy images, which effectively improves its haze-free feature representation ability without increasing network parameters. On the other hand, in the output of dehazing generator, four non-uniform image patches severely affected by residual haze are adaptively selected as input samples. The local discriminator could easily distinguish their hazy style, thereby further compelling the dehazing generator to suppress haze residues in such regions, thus enhancing its dehazing performance. Extensive experiments show that our PSD-ELGAN can achieve promising results and better generality across various datasets.

Correcting Optical Aberration via Depth-Aware Point Spread Functions.

Optical aberration is a ubiquitous degeneration in realistic lens-based imaging systems. Optical aberrations are caused by the differences in the optical path length when light travels through different regions of the camera lens with different incident angles. The blur and chromatic aberrations manifest significant discrepancies when the optical system changes. This work designs a transferable and effective image simulation system of simple lenses via multi-wavelength, depth-aware, spatially-variant four-dimensional point spread functions (4D-PSFs) estimation by changing a small amount of lens-dependent parameters. The image simulation system can alleviate the overhead of dataset collecting and exploiting the principle of computational imaging for effective optical aberration correction. With the guidance of domain knowledge about the image formation model provided by the 4D-PSFs, we establish a multi-scale optical aberration correction network for degraded image reconstruction, which consists of a scene depth estimation branch and an image restoration branch. Specifically, we propose to predict adaptive filters with the depth-aware PSFs and carry out dynamic convolutions, which facilitate the model's generalization in various scenes. We also employ convolution and self-attention mechanisms for global and local feature extraction and realize a spatially-variant restoration. The multi-scale feature extraction complements the features across different scales and provides fine details and contextual features. Extensive experiments demonstrate that our proposed algorithm performs favorably against state-of-the-art restoration methods.

Enhancing Visual Odometry with Estimated Scene Depth: Leveraging RGB-D Data with Deep Learning

Advances in visual odometry (VO) systems have benefited from the widespread use of affordable RGB-D cameras, improving indoor localization and mapping accuracy. However, older sensors like the Kinect v1 face challenges due to depth inaccuracies and incomplete data. This study compares indoor VO systems that use RGB-D images, exploring methods to enhance depth information. We examine conventional image inpainting techniques and a deep learning approach, utilizing newer depth data from devices like the Kinect v2. Our research highlights the importance of refining data from lower-quality sensors, which is crucial for cost-effective VO applications. By integrating deep learning models with richer context from RGB images and more comprehensive depth references, we demonstrate improved trajectory estimation compared to standard methods. This work advances budget-friendly RGB-D VO systems for indoor mobile robots, emphasizing deep learning’s role in leveraging connections between image appearance and depth data.

Digging into Depth and Color Spaces: A Mapping Constraint Network for Depth Super-Resolution

Scene depth super-resolution (DSR) poses an inherently ill-posed problem due to the extremely large space of one-to-many mapping functions from a given low-resolution (LR) depth map, which possesses limited depth information, to multiple plausible high-resolution (HR) depth maps. This characteristic renders the task highly challenging, as identifying an optimal solution becomes significantly intricate amidst this multitude of potential mappings. While simplistic constraints have been proposed to address the DSR task, the relationship between LR and HR depth maps and the color image has not been thoroughly investigated. In this paper, we introduce a novel mapping constraint network (MCNet) that incorporates additional constraints derived from both LR depth maps and color images. This integration aims to optimize the space of mapping functions and enhance the performance of DSR. Specifically, alongside the primary DSR network (DSRNet) dedicated to learning LR-to-HR mapping, we have developed an auxiliary degradation network (ADNet) that operates in reverse, generating the LR depth map from the reconstructed HR depth map to obtain depth features in LR space. To enhance the learning process of DSRNet in LR-to-HR mapping, we introduce two mapping constraints in LR space: 1) the cycle-consistent constraint, which offers additional supervision by establishing a closed loop between LR-to-HR and HR-to-LR mappings, and 2) the region-level contrastive constraint, aimed at reinforcing region-specific HR representations by explicitly modeling the consistency between LR and HR spaces. To leverage the color image effectively, we introduce a feature screening module (FSM) to adaptively fuse color features at different layers, which can simultaneously maintain strong structural context and suppress texture distraction through subspace generation and image projection. Comprehensive experimental results across synthetic and real-world benchmark datasets unequivocally demonstrate the superiority of our proposed method over state-of-the-art DSR methods. Our MCNet achieves an average MAD reduction of 3.7% and 7.5% over state-of-the-art DSR method for \(\times\) 8 and \(\times\) 16 cases on Milddleburry dataset, respectively, without incurring additional costs during inference.

Lightweight object detection in low light: Pixel-wise depth refinement and TensorRT optimization

Images captured in low-light conditions present significant challenges for accurate object detection due to factors such as high noise, poor illumination, and low contrast. In this study, we propose an innovative approach that combines pixel-wise depth refinement and TensorRT optimization to improve object detection performance and inference speed under challenging lighting scenarios. The Pixel-wise depth refinement utilizes a deep learning model to estimate the depth curve for each pixel in the low-light image, providing detailed scene depth information. We used the Lightweight YOLOv8 object detection model using TensorRT. This will reduce the model's computational complexity and memory footprint, enabling faster inference and real-time performance on resource-constrained devices. By combining these techniques, our approach aims to accomplish enhanced entity recognition precision in reduced-illumination environments while maintaining efficiency and real-time performance. We evaluate the effectiveness of our proposed modifications to the depth curve estimation block and the overall approach on benchmark datasets and demonstrate its potential for enhancing object detection capabilities in challenging low-light conditions. Furthermore, the experimental results show an improvement in the precision, recall, and mean average precisions (mAP), and also by using the TensorRT the inference speed is four times faster than the existing one.

Influence of scene aspect ratio and depth cues on verticality perception bias.

Perceiving verticality is crucial for accurate spatial orientation. Previous research has revealed that tilted scenes can bias verticality perception. Verticality perception bias can be represented as the sum of multiple periodic functions that play a role in the perception of visual orientation, where the specific factors affecting each periodicity remain uncertain. This study investigated the influence of the width and depth of an indoor scene on each periodic component of the bias. The participants were presented with an indoor scene showing a rectangular checkerboard room (Experiment 1), a rectangular aperture on the wall (Experiment 2), or a rectangular dotted room (Experiment 3), with various aspect ratios. The stimuli were presented with roll orientations ranging from 90° clockwise to 90° counterclockwise. The participants were asked to report their subjective visual vertical (SVV) perceptions. The contributions of 45°, 90°, and 180° periodicities to the SVV error were assessed by the weighted vector sum model. In Experiment 1, the periodic components of the SVV error increased with the aspect ratio. In Experiments 2 and 3, only the 90° component increased with the aspect ratio. These findings suggest that extended transverse surfaces may modulate the periodic components of verticality perception.

Monocular Depth and Ego-motion Estimation with Scale Based on Superpixel and Normal Constraints

3D perception in intelligent VR/AR and autonomous vehicles (AV) applications is critical and attracting significant attention. The self-supervised monocular depth and ego-motion estimation serves as a more intelligent learning approach that provides the required scene depth and location for 3D perception. However, the existing self-supervised learning methods suffer from scale ambiguity, boundary blur and imbalanced depth distribution, limiting the practical applications of VR/AR and AV. In this paper, we propose a new self-supervised learning framework based on superpixel and normal constraints to address these problems. Specifically, we formulate a novel 3D edge structure consistency loss to alleviate the boundary blur of depth estimation. To address the scale ambiguity of estimated depth and ego-motion, we propose a novel surface normal network for efficient camera height estimation. The surface normal network is composed of a deep fusion module and a full-scale hierarchical feature aggregation module. Meanwhile, to realize the global smoothing and boundary discriminability of the predicted normal map, we introduce a novel fusion loss which is based on the consistency constraints of the normal in edge domains and superpixel regions. Experiments are conducted on several benchmarks, and the results illustrate that the proposed approach outperforms the state-of-the-art methods in depth, ego-motion and surface normal estimation.

Bayesian neuromorphic imaging for single-photon LiDAR.

This paper proposes a Bayesian approach to enable single photon avalanche diode (SPAD) arrays to be used as pseudo event cameras that report changes in the scene. Motivated by the working principle of event cameras, which produce sparse events associated with light flux changes, we adopt a changepoint detection strategy to generate intensity and depth change event streams from direct time-of-flight (dToF) sequences measured by SPAD arrays. Although not our main goal, the algorithm also produces as a by-product, intensity and depth estimates. Unlike the output of passive event cameras that only correspond to light flux changes, the change events detected from the sequential dToFs can relate to changes in light flux and/or depth. The integration of the proposed Bayesian approach with single-photon LiDAR (SPL) systems provides a novel solution to achieve active neuromorphic 3D imaging that offers the advantages of significantly reduced output redundancy and in particular the capacity to report scene depth changes. For each pixel of the SPAD array, asynchronous events are generated by performing online Bayesian inference to detect changepoints and estimate the model parameters simultaneously from individual single-photon measurements. Experiments are conducted on synthetic data and real dToF measurements acquired by a 172×126 pixel SPAD camera to demonstrate the feasibility and efficiency of the proposed Bayesian approach.

LIASM-NRID: Constructing an atmospheric scattering model for low-light conditions and dehazing nighttime road images

Dehazing of nighttime road images holds significant practical relevance for autonomous driving and intelligent transportation systems operating in nocturnal conditions. To address the limitations of traditional atmospheric scattering models in accurately describing the imaging process in low illumination environments such as nighttime, this study constructs a low illumination environment atmospheric scattering model. Based on this newly constructed model, we propose an algorithm specifically designed for dehazing road images at nighttime. The proposed model incorporates an incident light attenuation factor, an additive noise factor, and a global compensation factor related to the scene's depth. The proposed nighttime road image dehazing algorithm firstly suppresses the interference of additive noise in the original image using wavelet decomposition and soft thresholding operation. Then, a transmittance-solving method containing the primary structural information of the image is designed based on the color attenuation linear model, and the solved transmittance is refined based on the interval gradient image texture structure method. Consequently, the global light map estimation process is formulated as an optimization problem to ensure the accuracy of the atmospheric light value calculation. In the final stages, a novel RGB color equalization method is introduced to address the issue of non-uniform incident light color bias in the dehazed images. Additionally, to eliminate the halo problem in the dehazed images, the adaptive histogram equalization algorithm is modified by incorporating a normalized gamma correction function. The experimental results show that the proposed nighttime road image dehazing algorithm exhibits robust performance and generality across various nighttime scenes. Compared with the representative algorithms for nighttime image dehazing, it can get more excellent dehazing results and has more outstanding performance in all numerical evaluation indexes.

H-SLAM: Hybrid direct–indirect visual SLAM

The recent success of hybrid methods in monocular odometry has led to many attempts to generalize the performance gains to hybrid monocular SLAM. However, most attempts fall short in several respects, with the most prominent issue being the need for two different map representations (local and global maps), with each requiring different, computationally expensive, and often redundant processes to maintain. Moreover, these maps tend to drift with respect to each other, resulting in contradicting pose and scene estimates, and leading to catastrophic failure. In this paper, we propose a novel approach that makes use of descriptor sharing to generate a single inverse depth scene representation. This representation can be used locally, queried globally to perform loop closure, and has the ability to re-activate previously observed map points after redundant points are marginalized from the local map, eliminating the need for separate map maintenance processes. The maps generated by our method exhibit no drift between each other, and can be computed at a fraction of the computational cost and memory footprint required by other monocular SLAM systems. Despite the reduced resource requirements, the proposed approach maintains its robustness and accuracy, delivering performance comparable to state-of-the-art SLAM methods (e.g., LDSO, ORB-SLAM3) on the majority of sequences from well-known datasets like EuRoC, KITTI, and TUM VI. The source code is available at: https://github.com/AUBVRL/fslam_ros_docker.

Dual-Frequency Lidar for Compressed Sensing 3D Imaging Based on All-Phase Fast Fourier Transform

Lidar, with its advantages of high measurement accuracy, fine angular resolution, and strong anti-interference capability, plays a pivotal role in the field of scene depth information acquisition. Traditional approaches to achieving lateral spatial resolution in imaging include raster scanning and array detectors. The former necessitates frequent scanning to acquire depth maps, resulting in time consumption and instability. The latter encounters challenges such as high dark count rates, pixel crosstalk, and excessive costs for obtaining high-resolution images using array detectors. The introduction of compressed sensing (CS) offers a novel perspective on realizing non-scanning three-dimensional imaging. In this context, we propose a novel three-dimensional imaging system that combines CS with coherent dual-frequency continuous-wave lidar and utilizes the all-phase fast Fourier transform to extract both amplitude and phase information. This system requires only M measurements, and through a reconstruction algorithm, it achieves the inversion of depth information for N-pixel scenes (M << N). Integrating cost-effective components such as digital micromirrors and single-point detectors, this affordable system accomplishes three-dimensional imaging of a single target. Notably, it significantly reduces the required number of measurements while concurrently ensuring enhanced eye safety and signal-to-noise ratio.

Sheared Epipolar Focus Spectrum for Dense Light Field Reconstruction.

This paper presents a novel technique for the dense reconstruction of light fields (LFs) from sparse input views. Our approach leverages the Epipolar Focus Spectrum (EFS) representation, which models the LF in the transformed spatial-focus domain, avoiding the dependence on the scene depth and providing a high-quality basis for dense LF reconstruction. Previous EFS-based LF reconstruction methods learn the cross-view, occlusion, depth and shearing terms simultaneously, which makes the training difficult due to stability and convergence problems and further results in limited reconstruction performance for challenging scenarios. To address this issue, we conduct a theoretical study on the transformation between the EFSs derived from one LF with sparse and dense angular samplings, and propose that a dense EFS can be decomposed into a linear combination of the EFS of the sparse input, the sheared EFS, and a high-order occlusion term explicitly. The devised learning-based framework with the input of the under-sampled EFS and its sheared version provides high-quality reconstruction results, especially in large disparity areas. Comprehensive experimental evaluations show that our approach outperforms state-of-the-art methods, especially achieves at most dB advantages in reconstructing scenes containing thin structures.

Monitoring Bioindication of Plankton through the Analysis of the Fourier Spectra of the Underwater Digital Holographic Sensor Data.

The study presents a bioindication complex and a technology of the experiment based on a submersible digital holographic camera with advanced monitoring capabilities for the study of plankton and its behavioral characteristics in situ. Additional mechanical and software options expand the capabilities of the digital holographic camera, thus making it possible to adapt the depth of the holographing scene to the parameters of the plankton habitat, perform automatic registration of the "zero" frame and automatic calibration, and carry out natural experiments with plankton photostimulation. The paper considers the results of a long-term digital holographic experiment on the biotesting of the water area in Arctic latitudes. It shows additional possibilities arising during the spectral processing of long time series of plankton parameters obtained during monitoring measurements by a submersible digital holographic camera. In particular, information on the rhythmic components of the ecosystem and behavioral characteristics of plankton, which can be used as a marker of the ecosystem well-being disturbance, is thus obtained.

A Multilayer Perceptron-Based Spherical Visual Compass Using Global Features.

This paper presents a visual compass method utilizing global features, specifically spherical moments. One of the primary challenges faced by photometric methods employing global features is the variation in the image caused by the appearance and disappearance of regions within the camera's field of view as it moves. Additionally, modeling the impact of translational motion on the values of global features poses a significant challenge, as it is dependent on scene depths, particularly for non-planar scenes. To address these issues, this paper combines the utilization of image masks to mitigate abrupt changes in global feature values and the application of neural networks to tackle the modeling challenge posed by translational motion. By employing masks at various locations within the image, multiple estimations of rotation corresponding to the motion of each selected region can be obtained. Our contribution lies in offering a rapid method for implementing numerous masks on the image with real-time inference speed, rendering it suitable for embedded robot applications. Extensive experiments have been conducted on both real-world and synthetic datasets generated using Blender. The results obtained validate the accuracy, robustness, and real-time performance of the proposed method compared to a state-of-the-art method.

Enhancing View Synthesis with Depth-Guided Neural Radiance Fields and Improved Depth Completion.

Neural radiance fields (NeRFs) leverage a neural representation to encode scenes, obtaining photorealistic rendering of novel views. However, NeRF has notable limitations. A significant drawback is that it does not capture surface geometry and only renders the object surface colors. Furthermore, the training of NeRF is exceedingly time-consuming. We propose Depth-NeRF as a solution to these issues. Specifically, our approach employs a fast depth completion algorithm to denoise and complete the depth maps generated by RGB-D cameras. These improved depth maps guide the sampling points of NeRF to be distributed closer to the scene's surface, benefiting from dense depth information. Furthermore, we have optimized the network structure of NeRF and integrated depth information to constrain the optimization process, ensuring that the termination distribution of the ray is consistent with the scene's geometry. Compared to NeRF, our method accelerates the training speed by 18%, and the rendered images achieve a higher PSNR than those obtained by mainstream methods. Additionally, there is a significant reduction in RMSE between the rendered scene depth and the ground truth depth, which indicates that our method can better capture the geometric information of the scene. With these improvements, we can train the NeRF model more efficiently and achieve more accurate rendering results.

Joint focusing method based on optimal image-distance interval for multi-image-distance imaging system

The multi-image-distance imaging system captures multiple images simultaneously with different focused depths, and uses image fusion to obtain an all-in-focus image for extending the depth of field (DoF). However, an image-distance adjustment strategy that takes into account the optimal DoF extension capability and imaging quality remains to be explored. In this paper, we propose a joint focusing method based on the optimal image-distance interval between the three sensors. First, we propose multi-sensor image-distance adjustment control strategies for different scenes. Then, theoretical analyses and calibration experiments of the optimal image-distance interval are conducted for continuous depth scenes. In order to quantitatively evaluate the DoF extension performance of the imaging system, a DoF estimation method based on point spread function measurement is proposed. Additionally, an evaluation metrics for extended DoF imaging based on the range and continuity of DoF is presented. We have developed a multi-image-distance electric focusing imaging system. Imaging experiments are conducted in various scenes to verify the effectiveness of the joint focusing method. The joint focusing method, based on the optimal image-distance interval, provides an image-distance adjustment strategy to achieve high-quality extended DoF imaging for multi-image-distance imaging systems.