High-resolution Depth Image Research Articles

DSRNet: Depth Super-Resolution Network guided by blurry depth and clear intensity edges

Although high resolution (HR) depth images are required in many applications such as virtual reality and autonomous navigation, their resolution and quality generated by consumer depth cameras fall short of the requirements. Existing depth upsampling methods focus on extracting multiscale features of HR color image to guide low resolution (LR) depth upsampling, thus causing blurry and inaccurate edges in depth. In this paper, we propose a depth super-resolution (SR) network guided by blurry depth and clear intensity edges, called DSRNet. DSRNet differentiates effective edges from a number of HR edges with the guidance of blurry depth and clear intensity edges. First, we perform global residual estimation based on an encoder–decoder architecture to extract edge structure from HR color image for depth SR. Then, we distinguish effective edges from HR edges in the decoder side with the guidance of LR depth upsampling. To maintain edges for depth SR, we use intensity edge guidance that extracts clear intensity edges from HR image. Finally, we use residual loss to generate accurate high frequency (HF) residual and reconstruct HR depth maps. Experimental results show that DSRNet successfully reconstructs depth edges in SR results as well as outperforms the state-of-the-art methods in terms of visual quality and quantitative measurements.11The proposed model with some test image pairs are available in https://github.com/lanhui-123/DSRNet.

Improved Upsampling Based Depth Image Super-Resolution Reconstruction

Constrained by current sensing technology, depth camera only acquires a low-resolution depth image that does not meet actual requirements. To solve this problem, this paper take a divide-and-conquer strategy to synthesize a high-resolution depth image from a low-resolution range image under the guidance of a registered high-resolution color image. Initially, the depth image is divided into planar areas and edge regions. For different zones, we exploit different methods to interpolate the missing depths. At planar area, the linear interpolation method is employed to perform upsampling. At edge region, a segmentation-separation upsampling method is used to interpolate the missing values. Then the upsampling result are refined on the Depth CNN that is built in this paper. We conduct extensive experiments on the benchmark database and real world data with various upsampling rates to illustrate the upsampling ability of our method. The comparison with classical super-resolution algorithms demonstrates that our upsampling algorithm achieves the best quality with fewer artifacts and our depth CNN outperforms the most state-of-the-art methods in terms of qualitative and quantitative evaluations.

Collaborative 3D face alignment and head pose estimation with frontal face constraint based on RGB and sparse depth

A collaborative multi-task learning framework is proposed for integrating 3D face alignment and head pose estimation using RGB and sparse depth with a frontal face constraint. Most previous works involve using dense depth images to exploit spatial information, but the acquisition of high-resolution dense depth images is not always available due to the limitation of the image capture device. Besides, the existing multi-task learning methods have the problem of insufficient learning since they have not bridged the semantic gap between both tasks. In this letter, integrating 3D face alignment and head pose estimation into a collaborative multi-task learning framework is proposed, which is supervised by a frontal face constraint. In addition, sparse depth is incorporated into the network to provide additional facial geometrical information. A UV position map is generated to conduct face alignment, and a head pose vector is output to predict the head orientation. The performance on AFLW2000-3D dataset is evaluated, and the minimum quantization errors are achieved for both tasks when compared with state-of-the-art methods, demonstrating the superiority of the proposed method.

High-resolution depth imaging with a small-scale SPAD array based on the temporal-spatial filter and intensity image guidance.

Currently single-photon avalanche diode (SPAD) arrays suffer from a small-scale pixel count, which makes it difficult to achieve high-resolution 3D imaging directly through themselves. We established a CCD camera-assisted SPAD array depth imaging system. Based on illumination laser lattice generated by a diffractive optical element (DOE), the registration of the low-resolution depth image gathered by SPAD and the high-resolution intensity image gathered by CCD is realized. The intensity information is used to guide the reconstruction of a resolution-enhanced depth image through a proposed method consisting of total generalized variation (TGV) regularization and temporal-spatial (T-S) filtering algorithm. Experimental results show that an increasement of 4 × 4 times for native depth image resolution is achieved and the depth imaging quality is also improved by applying the proposed method.

Optimisation of computer virtual image reconstruction method based on feature point matching

Traditional computer virtual image reconstruction methods have the problems of slow reconstruction speed and insufficient image clarity. To solve this problem, an optimisation method of computer virtual image reconstruction based on feature point matching is proposed. SURF operator is used to extract image feature points. After obtaining feature points, image feature points are matched according to TZNCC constraints. The virtual image is reconstructed by sparse method, and the high resolution depth image in virtual vision is represented by dictionary sparse linear combination. The sparse coefficient of the image is solved by alternating direction multiplier algorithm, and the problem of virtual image reconstruction is transformed into a problem of solving sparse signal, so that a better reconstruction effect can be obtained. The experimental results show that the proposed method has high speed and clarity of image reconstruction.

A large-scale dataset for indoor visual localization with high-precision ground truth

This article presents a challenging new dataset for indoor localization research. We have recorded the whole internal structure of Fengtai Wanda Plaza which is an area of over 15,800 m2 with a Navvis M6 device. The dataset contains 679 RGB-D panoramas and 2,664 query images collected by three different smartphones. In addition to the data, an aligned 3D point cloud is produced after the elimination of moving objects based on the building floorplan. Furthermore, a method is provided to generate corresponding high-resolution depth images for each panorama. By fixing the smartphones on the device using a specially designed bracket, six-degree-of-freedom camera poses can be calculated precisely. We believe it can give a new benchmark for indoor visual localization and the full dataset can be downloaded from http://vision.ia.ac.cn/Faculty/wgao/data_code/data_indoor_localizaiton/data_indoor_localization.htm

Three-Dimensional Laser Imaging with a Variable Scanning Spot and Scanning Trajectory

Traditional lidar scans the target with a fixed-size scanning spot and scanning trajectory. Therefore, it can only obtain the depth image with the same pixels as the number of scanning points. In order to obtain a high-resolution depth image with a few scanning points, we propose a scanning and depth image reconstruction method with a variable scanning spot and scanning trajectory. Based on the range information and the proportion of the area of each target (PAET) contained in the multi echoes, the region with multi echoes (RME) is selected and a new scanning trajectory and smaller scanning spot are used to obtain a finer depth image. According to the range and PAET obtained by scanning, the RME is segmented and filled to realize the super-resolution reconstruction of the depth image. By using this method, the experiments of two overlapped plates in space are carried out. By scanning the target with only forty-three points, the super-resolution depth image of the target with 160 × 160 pixels is obtained. Compared with the real depth image of the target, the accuracy of area representation (AOAR) and structural similarity (SSIM) of the reconstructed depth image is 99.89% and 98.94%, respectively. The method proposed in this paper can effectively reduce the number of scanning points and improve the scanning efficiency of the three-dimensional laser imaging system.

Color image multi-scale guided depth image super-resolution reconstruction

In order to obtain better super-resolution reconstruction results of depth images, this paper constructs a multi-scale color image guidance depth image super-resolution reconstruction convolutional neural network. In this paper, the multi-scale fusion method is used to realize the guidance of high resolution (HR) color image features to low resolution (LR) depth image features, which is beneficial to the restoration of image details. In the process of extracting features from LR depth images, a multiple receptive field residual block (MRFRB) is constructed to extract and fuse the features of different receptive fields, and then connect and fuse the features of each MRFRB output to obtain global fusion features. Finally, the HR depth image is obtained through sub-pixel convolution layer and global fusion features. The experimental results show that the super-resolution image obtained by this method alleviates the edge distortion and artifact problems, and has better visual effects.

Hetero-integration enables fast switching time-of-flight sensors for light detection and ranging

The time-of-flight (ToF) principle is a method used to measure distance and construct three-dimensional (3D) images by detecting the time or the phase difference between emitted and back-reflected optical flux. The ToF principle has been employed for various applications including light ranging and detection (LiDAR), machine vision and biomedical engineering; however, bulky system size and slow switching speed have hindered the widespread application of ToF technology. To alleviate these issues, a demonstration of hetero-integration of GaN-based high electron mobility transistors (HEMTs) and GaAs-based vertical cavity surface emitting lasers (VCSELs) on a single platform via a cold-welding method was performed. The hetero-integrated ToF sensors show superior switching performance when compared to silicon-transistor-based systems, miniaturizing size and exhibiting stable ranging and high-resolution depth-imaging. This hetero-integrated system of dissimilar material-based high-performance devices suggests a new pathway towards enabling high-resolution 3D imaging and inspires broader range application of heterogeneously integrated electronics and optoelectronics.

Super resolution of single depth image based on multi-dictionary learning with edge feature regularization

Currently, the acquisition and application of depth images are attracting a lot of attention thanks to the rapid development of 3D video. However, the current depth cameras cannot obtain high quality depth images due to the limitations in the imaging system. In this paper, to address this issue, we propose a scheme for single depth image super resolution based on multi-dictionary learning with edge regularization model. In the training stage, we focus on the edge information that represents the structure of the image, and extract the edge part, the low frequency and high frequency parts from the high resolution depth image set respectively. After that, three dictionaries are learned for the three parts with the constraint of the same sparse representation. In the image synthesis stage, we employ an edge-preserving regularization model as a reconstruction constraint to preserve the sharp structure, and reconstruct the depth image via the dictionaries learned from the training stage. Experimental results show that our proposed method can achieve good results in edge preservation, and both PSNR and SSIM values of the reconstructed depth images are superior to the state-of-art methods.

Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

In this paper, we propose a multi-frame depth super-resolution (SR) method based on L 1 data fidelity with the total variation regularization (TV-L 1 ) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L 1 model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L 1 data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct high-resolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise.

Long-range depth imaging using a single-photon detector array and non-local data fusion

The ability to measure and record high-resolution depth images at long stand-off distances is important for a wide range of applications, including connected and automotive vehicles, defense and security, and agriculture and mining. In LIDAR (light detection and ranging) applications, single-photon sensitive detection is an emerging approach, offering high sensitivity to light and picosecond temporal resolution, and consequently excellent surface-to-surface resolution. The use of large format CMOS (complementary metal-oxide semiconductor) single-photon detector arrays provides high spatial resolution and allows the timing information to be acquired simultaneously across many pixels. In this work, we combine state-of-the-art single-photon detector array technology with non-local data fusion to generate high resolution three-dimensional depth information of long-range targets. The system is based on a visible pulsed illumination system at a wavelength of 670 nm and a 240 × 320 array sensor, achieving sub-centimeter precision in all three spatial dimensions at a distance of 150 meters. The non-local data fusion combines information from an optical image with sparse sampling of the single-photon array data, providing accurate depth information at low signature regions of the target.

Automated segmentation of gravel particles from depth images of gravel-soil mixtures

We propose an image-based technique to measure the volume, weight and the size distribution of gravel particles in a gravel-soil mixture. The proposed method uses 3D scanning and a surface reconstruction algorithm to generate a high-resolution depth image, which is then used to accurately estimate the volume and weight of each gravel particle. The proposed method is evaluated on several gravel soil samples collected from 25 farming locations. The experimental results show that the proposed technique produces an accurate estimate of gravel volumes and gravel weights. It achieves a relative root mean square error of 4% for large gravel particles and an overall correlation of 0.99 with the ground truth, for the task of gravel volume estimation. For the estimation of gravel weight distribution, the proposed method can reach a low root mean square error of 0.54%. The rapid measurement of the full spectrum of coarse fragments in soil, using this method, is an advantage compared to the manual methods.

High-resolution depth imaging of long-offset 2D data using full-waveform inversion: The Malvinas Basin, Argentina

The Argentina Austral Malvinas survey comprises 13,784 km of 2D data extending from the shelf to the border with the Falkland Islands. The survey was acquired using a 12,000 m streamer and continuous recording technology and was processed through a comprehensive broadband prestack depth migration workflow focused on producing a high-resolution, high-fidelity data set. Source- and receiver-side deghosting to maximize the bandwidth of the data was an essential ingredient in the preprocessing. Following the broadband processing sequence, a depth-imaging workflow was implemented, with the initial model built using a time tomography approach. Several passes of anisotropic reflection tomography provided a significant improvement in the velocity model prior to full-waveform inversion (FWI). Using long offsets, FWI made use of additional information contained in the recorded wavefield, including the refracted and diving wave energy. FWI resolved more detailed velocity variations both in the shallow and deeper section and culminated in an improved seismic image.

Layer boundary evolution method for macular OCT layer segmentation.

Optical coherence tomography (OCT) is used to produce high resolution depth images of the retina and is now the standard of care for in-vivo ophthalmological assessment. It is also increasingly being used for evaluation of neurological disorders such as multiple sclerosis (MS). Automatic segmentation methods identify the retinal layers of the macular cube providing consistent results without intra- and inter-rater variation and is faster than manual segmentation. In this paper, we propose a fast multi-layer macular OCT segmentation method based on a fast level set method. Our framework uses contours in an optimized approach specifically for OCT layer segmentation over the whole macular cube. Our algorithm takes boundary probability maps from a trained random forest and iteratively refines the prediction to subvoxel precision. Evaluation on both healthy and multiple sclerosis subjects shows that our method is statistically better than a state-of-the-art graph-based method.

High-Quality 3D Reconstruction With Depth Super-Resolution and Completion

The 3D reconstruction is an important topic in computer vision with many applications, such as robotics and augmented reality. Since the raw depth images captured by consumer RGB-D cameras are often low resolution (LR), noisy, and incomplete. How to obtain high-quality 3D models with a consumer RGB-D camera is still a challenge for the existing systems. In this paper, we propose a new depth super-resolution and completion method implemented in a deep learning framework and build a high-quality 3D reconstruction system. We first improve the resolution of LR depth image with a depth super-resolution network and remove the outliers in high-resolution (HR) depth image based on gradient saliency. To further enhance the quality of HR depth image with the guide of HR color image, we learn surface normal and occlusion boundary images from the corresponding HR color image through two deep fully convolutional networks. In particular, the blurriness of HR color image is also detected and pixel-wise quantized. Finally, we obtain a completed HR depth image by optimizing the HR depth image with the surface normal, occlusion boundary, and color image blurriness. We have carried out qualitative and quantitative evaluations with baseline methods on public datasets. The experimental results demonstrate that our method has better performance both on single depth image enhancement and 3D reconstruction.

Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution.

Depth image super-resolution is a significant yet challenging task. In this paper, we introduce a novel deep color guided coarse-to-fine convolutional neural network (CNN) framework to address this problem. First, we present a datadriven filter method to approximate the ideal filter for depth image super-resolution instead of hand-designed filters. Based on large data samples, the filter learned is more accurate and stable for upsampling depth image. Second, we introduce a coarse-to-fine CNN to learn different sizes of filter kernels. In coarse stage, larger filter kernels are learned by CNN to achieve crude high-resolution depth image. As to fine stage, the crude high-resolution depth image is used as the input so that smaller filter kernels are learned to gain more accurate results. Benefit from this network, we can progressively recover the high frequency details. Third, we construct a color guidance strategy that fuses color difference and spatial distance for depth image upsampling. We revise the interpolated high-resolution depth image according to the corresponding pixels in highresolution color maps. Guided by color information, the depth of high-resolution image obtained can alleviate texture copying artifacts and preserve edge details effectively. Quantitative and qualitative experimental results demonstrate our state-of-the-art performance for depth map super-resolution.

Depth image super-resolution algorithm based on structural features and non-local means

The resolution and quality of the depth map captured by depth cameras are limited due to sensor hardware limitations, which becomes a roadblock for further computer vision applications. In order to solve this problem, we propose a new method to enhance low-resolution depth maps using high-resolution color images. The structural-aware term is introduced because of the availability of structural information in color images and the assumption of identical structural features within local neighborhoods of color images and depth images captured from the same scene. We integrate the structural-aware term with color similarity and depth similarity within local neighborhoods to design a local weighting filter based on structural features. To use non-local self-similarity of images, the local weighting filter is combined with the concept of non-local means, and then a non-local weighting filter based on structural features is designed. Some experimental results show that super-resolution depth image can be reconstructed well by the process of the non-local filter and the local filter based on structural features. The proposed method can reconstruct much better high-resolution depth images compared with previously reported methods.

Full waveform inversion with angle-dependent gradient preconditioning using wavefield decomposition

The main objective of the full waveform inversion (FWI) is to deliver a velocity model which produces accurate and high resolution depth images. An integral part of FWI is the iterative optimization process aiming at minimizing an objective function describing the misfit between the observed and modelled data with respect to the velocity model parameters. However, FWI commonly suffers from local minima due to errors in data fitting and the starting velocity model. In this study, we develop a methodology to enhance convergence and mitigate the risk of having solutions falling into local minima. We employ optical flow to decompose the source and residual receiver wavefields and map them into the subsurface angle domain. Having evaluated the propagation vectors of the respective cleaner wavefields, subsurface reflection and azimuth angles can be robustly computed. Using such information we can apply angle-dependent preconditioning within the inversion algorithm. During a conventional FWI workflow, larger angles (i.e., transmission wavepaths) are mainly responsible for long wavelength velocity anomalies whereas narrow angles (i.e., reflection wavepaths) influence short wavelength velocity anomalies. In the proposed FWI methodology, we apply an optimized weighting function based on these estimated subsurface reflection angles in order to weight certain angle directions needed for gradient estimation using in parameter updating. Following a hierarchical approach, allowing for larger angles at the early stage of the FWI application, we initially solve for the smooth and longer wavelength velocity updates. Relaxing the opening angle from large to mid to narrow, the gradient direction will first estimate long wavelength velocity model updates (transmission-based wavepaths), following by short wavelength velocity model updates (reflection-based wavepaths). Moreover, angle-dependent-based gradient preconditioning can reduce cycle skips and improve convergence during FWI iterations.

High-Resolution Ultrasonic Imaging of Artworks with Seismic Interferometry for Their Conservation and Restoration

ABSTRACTArtworks are an inseparable part of the cultural heritage of societies and provide us with a unique look at cultural developments through time and space. For the best possible conservation, it is paramount to know the constituent materials, condition, and construction techniques of the objects (e.g. painting on wood, fresco, sculpture). Such information is required not only for the surfaces of the objects, but also for the interiors; in the imaging discipline, this is known as depth imaging. Here, we introduce a new method for non-invasive depth imaging as an alternative to traditional non-invasive methods when the latter cannot be used to obtain the required information. We use ultrasonic transverse-wave transmission measurements and turn them into virtual reflection measurements. We achieve this by applying seismic interferometry with active sources. Obtaining reflection measurements by seismic interferometry allows us to apply an advanced imaging technique – prestack depth migration, as used in seismic exploration – to produce a high-resolution depth image of an object. We apply our method to ultrasonic data recorded on a mockup of a painting on a wooden support. We validate our method by comparing our results with an image from X-ray computed tomography.