Extract Depth Information Research Articles

Automatic hierarchical background virtualization method for monocular vision image based on depth information extraction

Automatic hierarchical background virtualization method for monocular vision image based on depth information extraction

High-speed all-in-focus 3D imaging method based on liquid lens focus scanning

A large open aperture in an optical system can capture high-resolution images but yields a shallow depth of field. To overcome this issue, we propose a method for retrofitting microscopy imaging systems by using a variable-focus liquid lens to achieve 3D focus scanning. Specifically, the focal length of the imaging system was changed by controlling the liquid lens, and a sequence of images was captured at different focal planes in milliseconds. The image scale and phase were corrected. Then the in-focus pixels were abstracted by employing the Laplacian operator. These pixels were marked in the index table. According to the index table, the focused parts of multiple images were merged, and an all-in-focus image was generated. At the same time, a depth map was constructed based on the image number in the index table and the extracted depth information. Additionally, we have optimized the image processing flow; the processing speed was improved to around 6.5 fps.

Depth-Guided Dehazing Network for Long-Range Aerial Scenes

Over the past few years, the applications of unmanned aerial vehicles (UAVs) have greatly increased. However, the decrease in clarity in hazy environments is an important constraint on their further development. Current research on image dehazing mainly focuses on normal scenes at close range or mid-range, while ignoring long-range scenes such as aerial perspective. Furthermore, based on the atmospheric scattering model, the inclusion of depth information is essential for the procedure of image dehazing, especially when dealing with images that exhibit substantial variations in depth. However, most existing models neglect this important information. Consequently, these state-of-the-art (SOTA) methods perform inadequately in dehazing when applied to long-range images. For the purpose of dealing with the above challenges, we propose the construction of a depth-guided dehazing network designed specifically for long-range aerial scenes. Initially, we introduce the depth prediction subnetwork to accurately extract depth information from long-range aerial images, taking into account the substantial variance in haze density. Subsequently, we propose the depth-guided attention module, which integrates a depth map with dehazing features through the attention mechanism, guiding the dehazing process and enabling the effective removal of haze in long-range areas. Furthermore, considering the unique characteristics of long-range aerial scenes, we introduce the UAV-HAZE dataset, specifically designed for training and evaluating dehazing methods in such scenarios. Finally, we conduct extensive experiments to test our method against several SOTA dehazing methods and demonstrate its superiority over others.

Optimizing Underwater Image Restoration and Depth Estimation with Light Field Images

Methods based on light field information have shown promising results in depth estimation and underwater image restoration. However, improvements are still needed in terms of depth estimation accuracy and image restoration quality. Previous work on underwater image restoration employed an image formation model (IFM) that overlooked the effects of light attenuation and scattering coefficients in underwater environments, leading to unavoidable color deviation and distortion in the restored images. Additionally, the high blurriness and associated distortions in underwater images make depth information extraction and estimation very challenging. In this paper, we refine the light propagation model and propose a method to estimate the attenuation and backscattering coefficients of the underwater IFM. We simplify these coefficients into distance-related functions and design a relationship between distance and the darkest channel to estimate the water coefficients, effectively suppressing color deviation and distortion in the restoration results. Furthermore, to increase the accuracy of depth estimation, we propose using blur cues to construct a cost for refocusing in the depth direction, reducing the impact of high signal-to-noise ratio environments on depth information extraction, and effectively enhancing the accuracy and robustness of depth estimation. Finally, experimental comparisons show that our method achieves more accurate depth estimation and image restoration closer to real scenes compared to state-of-the-art methods.

Depth-Guided Bilateral Grid Feature Fusion Network for Dehazing.

In adverse foggy weather conditions, images captured are adversely affected by natural environmental factors, resulting in reduced image contrast and diminished visibility. Traditional image dehazing methods typically rely on prior knowledge, but their efficacy diminishes in practical, complex environments. Deep learning methods have shown promise in single-image dehazing tasks, but often struggle to fully leverage depth and edge information, leading to blurred edges and incomplete dehazing effects. To address these challenges, this paper proposes a deep-guided bilateral grid feature fusion dehazing network. This network extracts depth information through a dedicated module, derives bilateral grid features via Unet, employs depth information to guide the sampling of bilateral grid features, reconstructs features using a dedicated module, and finally estimates dehazed images through two layers of convolutional layers and residual connections with the original images. The experimental results demonstrate the effectiveness of the proposed method on public datasets, successfully removing fog while preserving image details.

BSI-MVS: multi-view stereo network with bidirectional semantic information

The basic principle of multi-view stereo (MVS) is to perform 3D reconstruction by extracting depth information from multiple views. Most current SOTA MVS networks are based on Vision Transformer, which usually means expensive computational complexity. To reduce computational complexity and improve depth map accuracy, we propose a MVS network with Bidirectional Semantic Information (BSI-MVS). Firstly, we design a Multi-Level Spatial Pyramid module to generate multiple layers of feature map for extracting multi-scale information. Then we propose a 2D Bidirectional-LSTM module to capture bidirectional semantic information at different time steps in the horizontal and vertical directions, which contains abundant depth information. Finally, cost volumes are built based on various levels of feature maps to optimize the final depth map. We experiment on the DTU and BlendedMVS datasets. The result shows that our network, in terms of overall metrics, surpasses TransMVSNet, CasMVSNet, CVP-MVSNet, and AACVP-MVSNet respectively by 17.84%, 36.42%, 14.96%, and 4.86%, which also shows a noticeable performance enhancement in objective metrics and visualizations.

Are realistic details important for learning with visualizations or can depth cues provide sufficient guidance?

The optimal choice of the level of realism in instructional visualizations is a difficult task. Previous studies suggest that realism can overwhelm learners, but a growing body of research demonstrates that realistic details can enhance learning. In the first experiment (n = 107), it was assessed whether learning using realistic visualizations can be distracting and therefore particularly benefits from pre-training. Participants learned the anatomy of the parotid gland using labeled visualizations. While pre-training did not have an effect, a more realistic visualization enhanced learning compared to a schematic visualization. In the second experiment (n = 132), a schematic diagram was compared to a more realistic style featuring basic depth cues, and a highly realistic visualization containing a detailed surface. Regarding retention performance, no significant differences were found. However, an interesting pattern regarding subjective cognitive load ratings emerged: the schematic version received the highest cognitive load ratings, while the version featuring simplified shading was rated as least demanding. The version containing simplified depth cues also elicited lower cognitive load ratings than the detailed visualization. The two experiments demonstrate that fears concerning a detrimental effect of realistic details should not be over-generalized. While schematic visualizations may be easier to visually process in some cases, extracting depth information from contour drawings adds cognitive demands to a learning task. Thus, it is advisable that computer-generated visualizations contain at least simplified forms of shading, while the addition of details does not appear to have a strong positive effect.

DAE-Nest: A depth information extraction and enhancement fusion network for infrared and visible images

Fusion of information from infrared and visible images from different modalities has always been a hot research topic in the field of image processing. In order to solve the problem that existing fusion methods cannot fully utilize the deep information of images, this paper proposes a fusion algorithm DAE-Nest based on depth information extraction and enhancement. The algorithm consists of encoder-feature enhancement-fusion layer-decoder modules for unsupervised learning. At first, the improved coding structure is used to extract multi-scale features of the source image; secondly, the extracted features are data enhanced to fully utilize the deep image information; then, the enhanced features are fused according to the corresponding fusion strategy; finally, the decoder takes the input of fused features and reconstructs them into an information-rich fused image. DAE-Nest was subjected to corresponding fusion test experiments on the TNO and RoadScene data sets respectively and compared with 9 excellent fusion algorithms. In terms of subjective evaluation, DAE-Nest can make full use of the deep information in the source image, and the generated fusion image also conforms to the visual perception of the human eyes; in terms of objective evaluation indicators, DAE-Nest's average gradient and spatial frequency on the TNO data set, it achieved the best performance in the three evaluation indicators of structural similarity and structural similarity, and also achieved the best performance in the fusion quality evaluation index on the RoadScene data set. Therefore, the fusion method proposed in this paper has achieved excellent results both subjectively and objectively, and can effectively fuse infrared and visible images, laying a good foundation for subsequent visual tasks.

Depth of field extended integral imaging based on multi-depth fitting fusion

The traditional lens-based imaging display suffers from narrow depth of field (DoF), which is one of the primary drawbacks to restrict the commercial application of integral imaging (InIm). The limitation of DoF exists because locations of the reference plane and reconstructed 3D imagery are not identical. Herein, a DoF-extended method of rendering the elemental image array (EIA) based on multi-depth fitting fusion is proposed. In this method, depth stratification across the desired viewing depth range is implemented with depth maps for accurate extracting depth information and realizing high registration accuracy for image fusion, and then the multilayer parallax images (PIs) of different depth layers are obtained. Furthermore, an adaptive depth-fusion coding is proposed to establish effective iteration of depth-feature fitting for rendering the fusion EIA with the multilayer PIs, which uses the adaptive Gaussian-based weight matrix and the least squares method. As a result, the fusion EIA with multi-depth information, namely the iteratively weighted least squares solution of the EIA, is rendered to overcome the DoF limitation for InIm. The simulation and experimental presented 3D images with extended viewing depth range are demonstrated, verifying the feasibility of the proposed method.

Proton-exchange membrane fuel cell design for in-situ depth-sensitive X-ray absorption spectroscopy

We have built a proton exchange membrane hydrogen fuel cell optimized for angle-resolved X-ray absorption spectroscopy. This cell allows in-situ fluorescence measurements during electrochemical operation with minimal trade-offs in cell performance while reaching automotive current densities. The fluorescence signal can be collected from wide angles to extract depth information from the probed atomic species such as Pt, Co, and Ni, crucial to highly efficient FC. This cell is designed to assess the connection between the ionic drag/diffusion and the performance loss by following the real-time movement of the species through the membrane electrode assembly.

Assessment of out‐of‐plane structural defects using parallel laser line scanning system

AbstractA precise parallel laser line scanning system has been developed to assess the depth of out‐of‐plane structural defects on concrete surfaces. This system comprises a digital camera, dual line laser diodes, and positioning rigid arms that create a triangulation‐based setup. Laser lines are distorted when projected onto an out‐of‐plane defect. A new image processing algorithm has been devised to extract depth information from the distorted laser strips. Parallel laser lines are used to ensure that imaged laser strips do not intersect, thereby simplifying the depth assessment of defects at different distances and enabling the generation of defect profiles from a single image. The system has been validated through laboratory and field tests, demonstrating its effectiveness and accuracy. Compared to other noncontact measurement techniques, this system stands out due to its simplicity, cost‐effectiveness, efficiency, and superior accuracy for long‐range measurements, rendering it suitable for on‐site scanning of textureless uneven engineering objects.

Depth Completion in Autonomous Driving: Adaptive Spatial Feature Fusion and Semi-Quantitative Visualization

The safety of autonomous driving is closely linked to accurate depth perception. With the continuous development of autonomous driving, depth completion has become one of the crucial methods in this field. However, current depth completion methods have major shortcomings in small objects. To solve this problem, this paper proposes an end-to-end architecture with adaptive spatial feature fusion by encoder–decoder (ASFF-ED) module for depth completion. The architecture is built on the basis of the network architecture proposed in this paper, and is able to extract depth information adaptively with different weights on the specified feature map, which effectively solves the problem of insufficient depth accuracy of small objects. At the same time, this paper also proposes a depth map visualization method with a semi-quantitative visualization, which makes the depth information more intuitive to display. Compared with the currently available depth map visualization methods, this method has stronger quantitative analysis and horizontal comparison ability. Through experiments of ablation study and comparison, the results show that the method proposed in this paper exhibits a lower root-mean-squared error (RMSE) and better small object detection performance on the KITTI dataset.

Optical single-pixel volumetric imaging by three-dimensional light-field illumination

Three-dimensional single-pixel imaging (3D SPI) has become an attractive imaging modality for both biomedical research and optical sensing. 3D-SPI techniques generally depend on time-of-flight or stereovision principle to extract depth information from backscattered light. However, existing implementations for these two optical schemes are limited to surface mapping of 3D objects at depth resolutions, at best, at the millimeter level. Here, we report 3D light-field illumination single-pixel microscopy (3D-LFI-SPM) that enables volumetric imaging of microscopic objects with a near-diffraction-limit 3D optical resolution. Aimed at 3D space reconstruction, 3D-LFI-SPM optically samples the 3D Fourier spectrum by combining 3D structured light-field illumination with single-element intensity detection. We build a 3D-LFI-SPM prototype that provides an imaging volume of ∼390 × 390 × 3,800 μm3 and achieves 2.7-μm lateral resolution and better than 37-μm axial resolution. Its capability of 3D visualization of label-free optical absorption contrast is demonstrated by imaging single algal cells in vivo. Our approach opens broad perspectives for 3D SPI with potential applications in various fields, such as biomedical functional imaging.

A Monocular SLAM System Based on ResNet Depth Estimation

Currently, monocular SLAM (simultaneous location and mapping) systems cannot extract depth information from monocular cameras directly and require initialization to solve the problem of scale uncertainty. It is extremely difficult to reconstruct maps using such systems, and it is difficult to cope with scenarios that require navigation and obstacle avoidance. In order to solve the above problems, in this paper, a simple monocular depth estimation network framework is proposed. Transfer learning from a pre-trained ResNet is utilized for the encoding part of the framework and a CNN is used as the decoding part. Only a few training parameters and iterations are required to obtain fairly accurate depth information. At the same time, a similarity-based filter is used to denoise the surfels and improve the RGB-D SLAM system, which not only reduces the impact of the depth estimation error on the surfels, but also ensures the quality of the dense mapping. From the results of comparative experiments, it can be seen that the proposed monocular depth estimation network framework is better than current popular methods, and the associated SLAM system can achieve pose estimation and dense mapping tasks. As a monocular camera-based SLAM system, the proposed method is a promising and practical approach.

Use of ICEsat-2 and Sentinel-2 Open Data for the Derivation of Bathymetry in Shallow Waters: Case Studies in Sardinia and in the Venice Lagoon

Despite the high accuracy of conventional acoustic hydrographic systems, measurement of the seabed along coastal belts is still a complex problem due to the limitations arising from shallow water. In addition to traditional echo sounders, airborne LiDAR also suffers from high application costs, low efficiency, and limited coverage. On the other hand, remote sensing offers a practical alternative for the extraction of depth information, providing fast, reproducible, low-cost mapping over large areas to optimize and minimize fieldwork. Satellite-derived bathymetry (SDB) techniques have proven to be a promising alternative to supply shallow-water bathymetry data. However, this methodology is still limited since it usually requires in situ observations as control points for multispectral imagery calibration and bathymetric validation. In this context, this paper illustrates the potential for bathymetric derivation conducted entirely from open satellite data, without relying on in situ data collected using traditional methods. The SDB was performed using multispectral images from Sentinel-2 and bathymetric data collected by NASA’s ICESat-2 on two areas of relevant interest. To assess outcomes’ reliability, bathymetries extracted from ICESat-2 and derived from Sentinel-2 were compared with the updated and reliable data from the BathyDataBase of the Italian Hydrographic Institute.

Fault diagnosis strategy for few shot industrial process based on data augmentation and depth information extraction

AbstractIntelligent fault diagnosis method is an important tool for ensuring the stability of industrial processes. However, in the actual industrial process, forming a fault diagnosis model with good performance is difficult because of the complexity of feature extraction and the lack of labelled fault data. Data enhancement on the basis of the original data is important. To address this problem, this study proposes a method called self‐attention embedded generative adversarial network combined with a residual network (SAGAN‐ResNet). First, to address the lack of fault data, the data augmentation method consisting of the self‐attention embedded generator and discriminator is adopted. Then, to extract the features for better diagnosis performance, the residual network (ResNet) is introduced based on the augmented training dataset. A comparison of the proposed method with others shows that it has advantages in the case of complex process fault diagnosis with few‐shot industrial data.

Automatic depth map retrieval from digital holograms using a deep learning approach.

Information extraction from computer-generated holograms using learning-based methods is a topic that has not received much research attention. In this article, we propose and study two learning-based methods to extract the depth information from a hologram and compare their performance with that of classical depth from focus (DFF) methods. We discuss the main characteristics of a hologram and how these characteristics can affect model training. The obtained results show that it is possible to extract depth information from a hologram if the problem formulation is well-posed. The proposed methods are faster and more accurate than state-of-the-art DFF methods.

Link-RGBD: Cross-Guided Feature Fusion Network for RGBD Semantic Segmentation

Significant advances have been made in the development of deep convolutional networks for RGBD semantic segmentation. However, the full extraction and utilization of depth information to assist semantic segmentation remain challenging. In this study, we propose the Link-RGBD module to fuse RGB and depth features using an innovative interactive attention mechanism to augment the representations of the objects of interest. The attention map gotten from RGB and depth branch is regarded as a weight for value (v) in the other branch in order to achieve cross-fusion. This enhances the coordinates and feature information and augments the representations of the objects of interest. We evaluated the effectiveness of the proposed method and achieved competitive results on the challenging SUN RGB-D and NYUDv2 semantic segmentation benchmarks. Its pixel accuracy is 0.2% higher than that of DeepLabv3+, reaching 83.1% on the SUN RGB-D benchmark. Moreover, its mean IoU score is 0.3% higher than that of DeepLabv3+, reaching 49.5% on the NYUDv2 benchmark. We further constructed a robot test system to evaluate the performance of the proposed method in a real scenario. The results show that the proposed method can support the completion of the moving forward and grasping task.

Forecasting of Daily Outpatient Visits Based on Genetic Programming.

The forecasting of daily outpatient visits has significant practical implications in outpatient clinic operation management, not only contributing to guiding long-term resource planning and scheduling but also making tactical resolutions for short-term adjustments on special days such as holidays. We here in propose an effective genetic programming (GP)-based forecasting model to predict daily outpatient visits (OV) in a primary hospital. In the GP-based model, the holiday-based distance outlier mining algorithm was used to determine the holiday effect. In addition, solar terms were applied as the smallest unit to more accurately determine the impact of a change in the climate on the outpatient volume. A segmental learning strategy also was used to predict the daily outpatient volume for the time series data. The GP-based prediction could more effectively extract depth information from a finite training sample size and achieve a better performance for predicting daily outpatient visits, with lower root mean square error (RMSE) and higher coefficient of determination (R2) values, than the seasonal autoregressive integrated moving average (SARIMA) model in the time range of holidays and the holiday effect. GP-based model can achieve better prediction performance by overcoming the shortcomings of the SARIMA model. The results can be applied to support decision-making and planning of outpatient clinic resources, to help managers implement periodic scheduling of available resources on the basis of periodic features, and to perform proactive scheduling of additional resources.

A deep map transfer learning method for face recognition in an unrestricted smart city environment

Background and objectiveFace images often change in posture, Angle, expression, makeup, aging, lighting, and other aspects. The performance of many models with high accuracy in restricted scenes decreases sharply in unrestricted scenes. Therefore, the study of face recognition in unrestricted scenes is of great significance. In this paper, we proposed a face recognition algorithm based on depth map transfer learning to effectively recognize face images taken in unrestricted environment. MethodsFirstly, the deep network encoding and decoding structure based on dense feature fusion is used to extract depth information from face images. Secondly, depth information for migration learning is fused to improve the feature dimension of small sample data, thereby improving the problem of insufficient feature extraction training due to a small number of samples. Finally, in order to improve the classifier recognition performance in small sample machine learning, central loss is introduced into the traditional Softmax loss to improve the punishment supervision ability of the classification loss function.