Scene Depth Research Articles

Color-preserving visible and near-infrared image fusion for removing fog

With the unavailability of scene depth information, single-sensor dehazing methods based on deep learning or prior information do not effectively work in dense foggy scenes. An effective approach is to remove the dense fog by fusing visible and near-infrared images. However, the current dehazing algorithms based on near-infrared and visible images experience color distortion and information loss. To overcome these challenges, we proposed a color-preserving dehazing method that fuses near-infrared and visible images by introducing a dataset (VN-Haze) of visible and near-infrared images captured under hazy conditions. A two-stage image enhancement (TSE) method that can effectively rectify the color of visible images affected by fog was proposed to prevent the introduction of distorted color information. Furthermore, we proposed an adaptive luminance mapping (ALM) method to prevent color bias in fusion images caused by excessive differences in brightness between visible and near-infrared images that occur in vegetation areas. The proposed visible-priority fusion strategy reasonably allocates weights for visible and near-infrared images, minimizing the loss of important features in visible images. Compared with existing dehazing algorithms, the proposed algorithm generates images with natural colors and less distortion and retains important visible information. Moreover, it demonstrates remarkable performance in objective evaluations.

Multiscale Cross-Connected Dehazing Network With Scene Depth Fusion.

In this article, we propose a multiscale cross-connected dehazing network with scene depth fusion. We focus on the correlation between a hazy image and the corresponding depth image. The model encodes and decodes the hazy image and the depth image separately and includes cross connections at the decoding end to directly generate a clean image in an end-to-end manner. Specifically, we first construct an input pyramid to obtain the receptive fields of the depth image and the hazy image at multiple levels. Then, we add the features of the corresponding dimensions in the input pyramid to the encoder. Finally, the two paths of the decoder are cross-connected. In addition, the proposed model uses wavelet pooling and residual channel attention modules (RCAMs) as components. A series of ablation experiments shows that the wavelet pooling and RCAMs effectively improve the performance of the model. We conducted extensive experiments on multiple dehazing datasets, and the results show that the model is superior to other advanced methods in terms of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and subjective visual effects. The source code and supplementary are available at https://github.com/CCECfgd/MSCDN-master.

Depth Image Completion through Iterative Low-Pass Filtering

This study introduces a spatial-modulated approach designed to recover missing data in in-depth images. Typically, commercial-grade RGB-D cameras utilize structured light or time-of-flight techniques for capturing scene depth. However, these conventional methods encounter difficulties in acquiring depth data from glossy, transparent, or low-reflective surfaces. Additionally, they are prone to interference from broad-spectrum light sources, resulting in defective areas in the captured data. The generation of dense data is further compromised by the influence of noise. In response to these challenges, we implemented an iterative low-pass filter in the frequency domain, effectively mitigating noise and restoring high-quality depth data across all surfaces. To assess the efficacy of our method, deliberate introduction of significant noise and induced defects in the generated depth images was performed. The experimental results unequivocally demonstrate the promising accuracy, precision, and noise-resilient capabilities of our approach. Our implementation is publicly available on the project’s webpage.

Learning Spherical Radiance Field for Efficient 360° Unbounded Novel View Synthesis.

Novel view synthesis aims at rendering any posed images from sparse observations of the scene. Recently, neural radiance fields (NeRF) have demonstrated their effectiveness in synthesizing novel views of a bounded scene. However, most existing methods cannot be directly extended to 360° unbounded scenes where the camera orientations and scene depths are unconstrained with large variations. In this paper, we present a spherical radiance field (SRF) for efficient novel view synthesis in 360° unbounded scenes. Specifically, we represent a 3D scene as multiple concentric spheres with different radii. In particular, each sphere encodes its corresponding layered scene into implicit representations and is parameterized with an equirectangular projection image. A shallow multi-layer perceptron (MLP) is then used to infer the density and color from these sphere representations for volume rendering. Moreover, an occupancy grid is introduced to cache the density field and guide the ray sampling, which accelerates the training and rendering procedures by reducing the number of samples along the ray. Experiments show that our method can well fit 360° unbounded scenes and produces state-of-the-art results on three benchmark datasets with less than 30 minutes of training time on a 3090 GPU, surpassing Mip-NeRF 360 with a 400× speedup. In addition, our method achieves competitive performance in terms of both accuracy and efficiency on a bounded dataset. Project page: https://minglin-chen.github.io/SphericalRF.

Deep Depth from Focal Stack with Defocus Model for Camera-Setting Invariance

We propose deep depth from focal stack (DDFS), which takes a focal stack as input of a neural network for estimating scene depth. Defocus blur is a useful cue for depth estimation. However, the size of the blur depends on not only scene depth but also camera settings such as focus distance, focal length, and f-number. Current learning-based methods without any defocus models cannot estimate a correct depth map if camera settings are different at training and test times. Our method takes a plane sweep volume as input for the constraint between scene depth, defocus images, and camera settings, and this intermediate representation enables depth estimation with different camera settings at training and test times. This camera-setting invariance can enhance the applicability of DDFS. The experimental results also indicate that our method is robust against a synthetic-to-real domain gap.

Highway Visibility Estimation in Foggy Weather via Multi-Scale Fusion Network.

Poor visibility has a significant impact on road safety and can even lead to traffic accidents. The traditional means of visibility monitoring no longer meet the current needs in terms of temporal and spatial accuracy. In this work, we propose a novel deep network architecture for estimating the visibility directly from highway surveillance images. Specifically, we employ several image feature extraction methods to extract detailed structural, spectral, and scene depth features from the images. Next, we design a multi-scale fusion network to adaptively extract and fuse vital features for the purpose of estimating visibility. Furthermore, we create a real-scene dataset for model learning and performance evaluation. Our experiments demonstrate the superiority of our proposed method to the existing methods.

Restoration of Binocular Images Degraded by Optical Scattering through Estimation of Atmospheric Coefficients.

A binocular vision-based approach for the restoration of images captured in a scattering medium is presented. The scene depth is computed by triangulation using stereo matching. Next, the atmospheric parameters of the medium are determined with an introduced estimator based on the Monte Carlo method. Finally, image restoration is performed using an atmospheric optics model. The proposed approach effectively suppresses optical scattering effects without introducing noticeable artifacts in processed images. The accuracy of the proposed approach in the estimation of atmospheric parameters and image restoration is evaluated using synthetic hazy images constructed from a well-known database. The practical viability of our approach is also confirmed through a real experiment for depth estimation, atmospheric parameter estimation, and image restoration in a scattering medium. The results highlight the applicability of our approach in computer vision applications in challenging atmospheric conditions.

DUEL: Depth visUal Ego-motion Learning for autonomous robot obstacle avoidance

Reliable obstacle avoidance, which is essential for safe autonomous robot interaction with the real world, raises various challenges such as difficulties with obstacle perception and latent factor cognition impacting multi-modal obstacle avoidance. In this paper, we propose a Depth visUal Ego-motion Learning (DUEL) model, consisting of a cognitive generation network, a policy decision network and a potential partition network, to learn autonomous obstacle avoidance from expert policies. The DUEL model takes advantage of binocular vision to perceive scene depth. This serves as the input to the cognitive generation network which generates obstacle avoidance policies by maximizing its causal entropy. The policy decision network then optimizes the generation of the policies referring to expert policies. The generated obstacle avoidance policies are simultaneously transferred to the potential partition network to capture the latent factors contained within expert policies and perform multi-modal obstacle avoidance. These three core networks iteratively optimize the multi-modal policies relying on causal entropy and mutual information theorems, which are proven theoretically. Experimental comparisons with state-of-the-art models on 7 metrics demonstrate the effectiveness of the DUEL model. It achieves the best performance with an average ADE (Average Displacement Error) of 0.29 and average FDE (Final Displacement Error) of 0.55 across five different scenarios. Results show that the DUEL model can maintain an average obstacle avoidance success rate of 97% for both simulated and real world scenarios with multiple obstacles, demonstrating its success at capturing latent factors from expert policies. Our source codes are available at https://github.com/ACoTAI/DUEL.

Underwater Image Quality Assessment from Synthetic to Real-world: Dataset and Objective Method

The complicated underwater environment and lighting conditions lead to severe influence on the quality of underwater imaging, which tends to impair underwater exploration and research. To effectively evaluate the quality of underwater images, an underwater image quality assessment dataset is constructed from synthetic to real-world, and then a new objective underwater image assessment method based on the characteristics of the underwater imaging is proposed (UICQA). Specifically, to address the lack of a publicly available datasets and more accurately quantify the quality of underwater images, a subjective underwater image quality assessment dataset from synthetic to real-world underwater images, named USRD, is constructed. Considering that the transmission map can effectively reflect the characteristics of the underwater imaging, statistical features are effectively extracted from the transmission map for distinguishing underwater images of different quality. Further, considering that the transmission map negatively correlates with scene depth, a local-to-global transmission map weighted contrast feature is constructed. Additionally, the color features of human perception and texture features based on fractal dimensions are proposed. Finally, the experimental results show that the proposed UICQA method exhibits the highest correlation with ground truth scores compared to state-of-the-art UIQA methods.

Fast Rolling Shutter Correction in the Wild.

This paper addresses the problem of rolling shutter correction (RSC) in uncalibrated videos. Existing works remove rolling shutter (RS) distortion by explicitly computing the camera motion and depth as intermediate products, followed by motion compensation. In contrast, we first show that each distorted pixel can be implicitly rectified back to the corresponding global shutter (GS) projection by rescaling its optical flow. Such a point-wise RSC is feasible with both perspective and non-perspective cases without the pre-knowledge of the camera used. Besides, it allows a pixel-wise varying direct RS correction (DRSC) framework that handles locally varying distortion caused by various sources, such as camera motion, moving objects, and even highly varying depth scenes. More importantly, our approach is an efficient CPU-based solution that enables undistorting RS videos in real-time (40fps for 480p). We evaluate our approach across a broad range of cameras and video sequences, including fast motion, dynamic scenes, and non-perspective lenses, demonstrating the superiority of our proposed approach over state-of-the-art methods in both effectiveness and efficiency. We also evaluated the ability of the RSC results to serve for downstream 3D analysis, such as visual odometry and structure-from-motion, which verifies preference for the output of our algorithm over other existing RSC methods.

Robust single-photon 3D imaging based on full-scale feature integration and intensity edge guidance

Active single-photon light detection and ranging (LiDAR) imaging technology adopting emerging single-photon avalanche diode (SPAD) has been used extensively for 3D imaging of complex scenes. However, the different signal-to-noise data acquired by SPAD under complex scenes and the larger pixel size of the SPAD pose a great challenge for robustly estimating the scene depth with rich detailed information. In this paper, we have designed a two-stage network including a multi-scale encoder-decoder sub-network and an intensity-guided edge refinement sub-network using a sensor fusion strategy to improve the robustness of the depth estimation network. In the first stage of the sub-network, we use the UNet3+ network to integrate fine-grained detail information and coarse-grained semantic information from full-scale through skip connections across scales on the decoder. In the meantime, the deep boosting network is added to the encoder to extract rich contextual information from the spatial-temporal domain to alleviate semantic gaps between the encoder and decoder. Moreover, in the second stage of the sub-network, we adopt an inverted y-shaped backbone network with the same structure between the encoder and decoder coupled with the same weight at both encoders. The purpose of this design is to extract structural similarity features with spatial consistency from intensity and depth to compensate for the missing detail of reconstructed depth by the first stage sub-network. The experimental results conducted on both simulated and real-world datasets demonstrate that our network outperforms previous networks. In addition, networks are trained on single or multiple signal-to-background ratio datasets to estimate the real-world data respectively, and the results show that our proposed network is more robust.

Features of the Application of Coherent Noise Suppression Methods in the Digital Holography of Particles

The paper studies the influence of coherent noises on the quality of images of particles reconstructed from digital holograms. Standard indicators (for example, signal-to-noise ratio) and such indicators as the boundary contrast and boundary intensity jump previously proposed by the authors are used to quantify the image quality. With the use of these parameters, for examples of some known methods of suppressing coherent noises in a holographic image (eliminating the mutual influence of virtual and real images in in-line holography, and time averaging), the features and ranges of applicability of such correction were determined. It was shown that the use of the complex field amplitude reconstruction method based on the Gerchberg–Saxton algorithm and the spatial-frequency method improves the quality of determining the particle image boundary (by boundary intensity jump) starting from the distance between a hologram and a particle, which is about twice the Rayleigh distance. In physical experiments with model particles, averaging methods were studied to suppress non-stationary coherent noises (speckles). It was also shown that averaging over three digital holograms or over three holographic images is sufficient to provide a quality of particle image boundary suitable for particle recognition. In the case of multiple scattering, when it is necessary to impose a limit on the working volume length (depth of scene) of the holographic camera, the paper provides estimates that allow selecting the optimal working volume length. The estimates were made using the example of a submersible digital holographic camera for plankton studies.

Spatial Scene Memories Are Biased Towards a Fixed Amount of Semantic Information.

Scene memory has known spatial biases. Boundary extension is a well-known bias whereby observers remember visual information beyond an image's boundaries. While recent studies demonstrate that boundary contraction also reliably occurs based on intrinsic image properties, the specific properties that drive the effect are unknown. This study assesses the extent to which scene memory might have a fixed capacity for information. We assessed both visual and semantic information in a scene database using techniques from image processing and natural language processing, respectively. We then assessed how both types of information predicted memory errors for scene boundaries using a standard rapid serial visual presentation (RSVP) forced error paradigm. A linear regression model indicated that memories for scene boundaries were significantly predicted by semantic, but not visual, information and that this effect persisted when scene depth was considered. Boundary extension was observed for images with low semantic information, and contraction was observed for images with high semantic information. This suggests a cognitive process that normalizes the amount of semantic information held in memory.

A multimodal feature fusion image dehazing method with scene depth prior

AbstractCurrent dehazing networks usually only learn haze features in a single‐image colour space and often suffer from uneven dehazing, colour, and edge degradation when confronted with different scales of ground objects in the depth space of the scene. The authors propose a multimodal feature fusion image dehazing method with scene depth prior based on a decoder–encoder backbone network. The multimodal feature fusion module was first designed. In this module, affine transformation and polarized self‐attention mechanism are used to realize the fusion of image colour and depth prior feature, to improve the representation ability of the model for different scale ground haze feature in‐depth space. Then, the feature enhancement module (FEM) is added, and deformable convolution and difference convolution methods are used to enhance the representation ability of the model for the geometric and texture feature of the ground objects. The publicly available dehazing datasets are used for comparison and ablation experiments. The results show that compared with the existing classical dehazing networks, the peak signal‐to‐noise ratio (PSNR) and SSIM of the authors’ proposed method have been significantly improved, have a more uniform dehazing effect in different depth spaces, and maintain the colour and edge details of the ground objects very well.

Reverse gamma correction based GARCH model for underwater image dehazing and detail exposure

Underwater imaging poses significant challenges as water alters the behavior of light in comparison to air or vacuum. Therefore, it is crucial to effectively utilize the unique characteristics of unclear edges in hazy underwater images to achieve high-performance results in real-time applications. In this paper, we exploit such features as edges and visual perception in underwater haze images. To achieve this, we estimate the true transmission of the image by enhancing the visibility of discontinuous edges using the reverse gamma correction based on the Generalized Autoregressive Conditional Heteroscedasticity (GARCH) model. The parameters of the GARCH model are defined by the local and global pixel dynamics in adjacent neighborhoods, which improves the color orientation of the image while preserving image details. Additionally, we perform deseasoning to separate pixels while maintaining the natural contours of interactions between them. By considering the volatility of the RGB color channels, we predict the variance of the pixels based on the difference of the deseasoned pixels, which improves pixel intensity and allows for scene depth estimation. While estimating gamma correction and global ambient light, we deseasonalize image pixels based on their colors, enhancing the color of the final dehazed images. Using the greedy algorithm with a Convolutional Neural Network (CNN), our proposed method outperforms commonly used state-of-the-art methods.

Enhancement and Optimization of Underwater Images and Videos Mapping

Underwater images tend to suffer from critical quality degradation, such as poor visibility, contrast reduction, and color deviation by virtue of the light absorption and scattering in water media. It is a challenging problem for these images to enhance visibility, improve contrast, and eliminate color cast. This paper proposes an effective and high-speed enhancement and restoration method based on the dark channel prior (DCP) for underwater images and video. Firstly, an improved background light (BL) estimation method is proposed to estimate BL accurately. Secondly, the R channel’s transmission map (TM) based on the DCP is estimated sketchily, and a TM optimizer integrating the scene depth map and the adaptive saturation map (ASM) is designed to refine the afore-mentioned coarse TM. Later, the TMs of G–B channels are computed by their ratio to the attenuation coefficient of the red channel. Finally, an improved color correction algorithm is adopted to improve visibility and brightness. Several typical image-quality assessment indexes are employed to testify that the proposed method can restore underwater low-quality images more effectively than other advanced methods. An underwater video real-time measurement is also conducted on the flipper-propelled underwater vehicle-manipulator system to verify the effectiveness of the proposed method in the real scene.

A novel collision warning system based on the visual road environment schema: An examination from vehicle and driver characteristics

Drivers pay unequal attention to different road environmental elements and visual fields, which greatly influences their driving behavior. However, existing collision warning systems ignore these visual characteristics of drivers, which limits the performance of collision warning systems. Therefore, this study proposes a novel collision warning system based on the visual road environment schema, in order to enhance the support for avoiding potential dangers in objects and areas that are easily overlooked by the drivers’ vision. To capture the above visual characteristics of drivers, the visual road environment schema that consists of the semantic layer, the scene depth layer, the sensitive layer, and the visual field layer is established by using several different deep neural networks, which realizes the recognition, quantization, and analysis of the road environment from the drivers’ visual perspective. The effectiveness of the novel collision warning system is verified by the driving simulation experiment from six indicators, including warning distance, maximum lateral acceleration, maximum longitudinal deceleration, minimum collision time, reaction time, and heart rate. Additionally, a grey target decision-making model is built to comprehensively evaluate the system. The results show that compared with the traditional collision warning system, the novel collision warning system proposed in this study performs significantly better and can discover potential dangers earlier, give timely warnings, enhance the vehicles’ lateral stability and driving comfort, shorten reaction time, and relieve the drivers’ nervousness. By integrating the drivers’ visual characteristics into the collision warning system, this study could help to optimize the existing collision warning system and promote the mutual understanding between intelligent vehicles and human drivers.

Unsupervised Monocular Depth and Camera Pose Estimation with Multiple Masks and Geometric Consistency Constraints.

This paper presents a novel unsupervised learning framework for estimating scene depth and camera pose from video sequences, fundamental to many high-level tasks such as 3D reconstruction, visual navigation, and augmented reality. Although existing unsupervised methods have achieved promising results, their performance suffers in challenging scenes such as those with dynamic objects and occluded regions. As a result, multiple mask technologies and geometric consistency constraints are adopted in this research to mitigate their negative impacts. Firstly, multiple mask technologies are used to identify numerous outliers in the scene, which are excluded from the loss computation. In addition, the identified outliers are employed as a supervised signal to train a mask estimation network. The estimated mask is then utilized to preprocess the input to the pose estimation network, mitigating the potential adverse effects of challenging scenes on pose estimation. Furthermore, we propose geometric consistency constraints to reduce the sensitivity of illumination changes, which act as additional supervised signals to train the network. Experimental results on the KITTI dataset demonstrate that our proposed strategies can effectively enhance the model's performance, outperforming other unsupervised methods.

Perceptual Assessment of Image and Depth Quality of Dynamically Depth-Compressed Scene for Automultiscopic 3D Display.

This article discusses the depth range which automultiscopic 3D (A3D) displays should reproduce for ensuring an adequate perceptual quality of substantially deep scenes. These displays usually need sufficient depth reconstruction capabilities covering the whole scene depth, but due to the inherent hardware restriction of these displays this is often difficult, particularly for showing deep scenes. Previous studies have addressed this limitation by introducing depth compression that contracts the scene depth into a smaller depth range by modifying the scene geometry, assuming that the scenes were represented as CG data. The previous results showed that reconstructing only a physical depth of 1 m is needed to show scenes with much deeper depth and without large perceptual quality degradation. However, reconstructing a depth of 1 m is still challenging for actual A3D displays. In this study, focusing on a personal viewing situation, we introduce a dynamic depth compression that combines viewpoint tracking with the previous approach and examines the extent to which scene depths can be compressed while keeping the original perceptual quality. Taking into account the viewer's viewpoint movements, which were considered a cause of unnaturalness in the previous approach, we performed an experiment with an A3D display simulator and found that a depth of just 10 cm was sufficient for showing deep scenes without inducing a feeling of unnaturalness. Next, we investigated whether the simulation results were valid even on a real A3D display and found that the dynamic approach induced better perceptual quality than the static one even on the real A3D display and that it had a depth enhancing effect without any hardware updates. These results suggest that providing a physical depth of 10 cm on personalized A3D displays is general enough for showing any deeper 3D scenes with appealing subjective quality.

DPSNet: Multitask Learning Using Geometry Reasoning for Scene Depth and Semantics.

Multitask joint learning technology continues gaining more attention as a paradigm shift and has shown promising performance in many applications. Depth estimation and semantic understanding from monocular images emerge as a challenging problem in computer vision. While the other joint learning frameworks establish the relationship between the semantics and depth from stereo pairs, the lack of learning camera motion renders the frameworks that fail to model the geometric structure of the image scene. We make a further step in this article by proposing a multitask learning method, namely DPSNet, which can jointly perform depth and camera pose estimation and semantic scene segmentation. Our core idea for depth and camera pose prediction is that we present the rigid semantic consistency loss to overcome the limitation of moving pixels from image reconstruction technology and further infer the segmentation of moving instances based on them. In addition, our proposed model performs semantic segmentation by reasoning the geometric correspondences between the pixel semantic outputs and the semantic labels at multiscale resolutions. Experiments on open-source datasets and a video dataset captured on a micro-smart car show the effectiveness of each component of DPSNet, and DPSNet achieves state-of-the-art results in all three tasks compared with the best popular methods. All our models and code are available at https://github.com/jn-z/DPSNet: Multitask Learning Using Geometry Reasoning for Scene Depth and semantics.