Layered Depth Image Research Articles

End-to-end learning of 3D phase-only holograms for holographic display

Computer-generated holography (CGH) provides volumetric control of coherent wavefront and is fundamental to applications such as volumetric 3D displays, lithography, neural photostimulation, and optical/acoustic trapping. Recently, deep learning-based methods emerged as promising computational paradigms for CGH synthesis that overcome the quality-runtime tradeoff in conventional simulation/optimization-based methods. Yet, the quality of the predicted hologram is intrinsically bounded by the dataset’s quality. Here we introduce a new hologram dataset, MIT-CGH-4K-V2, that uses a layered depth image as a data-efficient volumetric 3D input and a two-stage supervised+unsupervised training protocol for direct synthesis of high-quality 3D phase-only holograms. The proposed system also corrects vision aberration, allowing customization for end-users. We experimentally show photorealistic 3D holographic projections and discuss relevant spatial light modulator calibration procedures. Our method runs in real-time on a consumer GPU and 5 FPS on an iPhone 13 Pro, promising drastically enhanced performance for the applications above.

A Study on H.264/AVC Video Compression Standard of Multi-view Image Expressed by Layered Depth Image

A Study on H.264/AVC Video Compression Standard of Multi-view Image Expressed by Layered Depth Image

One shot 3D photography

3D photography is a new medium that allows viewers to more fully experience a captured moment. In this work, we refer to a 3D photo as one that displays parallax induced by moving the viewpoint (as opposed to a stereo pair with a fixed viewpoint). 3D photos are static in time, like traditional photos, but are displayed with interactive parallax on mobile or desktop screens, as well as on Virtual Reality devices, where viewing it also includes stereo. We present an end-to-end system for creating and viewing 3D photos, and the algorithmic and design choices therein. Our 3D photos are captured in a single shot and processed directly on a mobile device. The method starts by estimating depth from the 2D input image using a new monocular depth estimation network that is optimized for mobile devices. It performs competitively to the state-of-the-art, but has lower latency and peak memory consumption and uses an order of magnitude fewer parameters. The resulting depth is lifted to a layered depth image, and new geometry is synthesized in parallax regions. We synthesize color texture and structures in the parallax regions as well, using an inpainting network, also optimized for mobile devices, on the LDI directly. Finally, we convert the result into a mesh-based representation that can be efficiently transmitted and rendered even on low-end devices and over poor network connections. Altogether, the processing takes just a few seconds on a mobile device, and the result can be instantly viewed and shared. We perform extensive quantitative evaluation to validate our system and compare its new components against the current state-of-the-art.

Free-Floating Support Structure Generation

To print a 3D model with a 3D SLA printer, support structures are required for areas such as overhangs and islands. If there is no support structure below islands, the solidified material will float around in the vat. Further deposition in the areas will sink as well, resulting in a printing failure. This issue is called the floating island problem. To resolve this problem, two different approaches to detect support points have been developed. One is a slice-based approach that pre-slices the given 3D model and compares all sets of two consecutive layers; this is computationally inefficient. The other is a mesh-based approach that investigates primitives such as points, edges, and faces and samples the clusters; this method, however, tends to miss the local minimums and causes the problem.In this paper, we propose a novel method to create support structures. This method does not lead to the floating island problem, in contrast to the mesh-based approach. Besides, this method does not need pre-slice 3D models, nor does it need to compare slices. It first converts a 3D model into a geometric representation called a layered depth image (LDI). It then creates a directed graph from the LDI to describe the delivery path of self-support and identifies efficient support points to prevent material waste. The structures can be reinforced by considering the volume as well as the geometry of the model. They are also reinforced by adding connections between anchors to increase stability. This method is compared to well-known commercial slicers. The result proves that this method is computationally efficient and never causes floating island problems.

Peeking behind objects: Layered depth prediction from a single image

While conventional depth estimation can infer the geometry of a scene from a single RGB image, it fails to estimate scene regions that are occluded by foreground objects. This limits the use of depth prediction in augmented and virtual reality applications, that aim at scene exploration by synthesizing the scene from a different vantage point, or at diminished reality. To address this issue, we shift the focus from conventional depth map prediction to the regression of a specific data representation called Layered Depth Image (LDI), which contains information about the occluded regions in the reference frame and can fill in occlusion gaps in case of small view changes. We propose a novel approach based on Convolutional Neural Networks (CNNs) to jointly predict depth maps and foreground separation masks used to condition Generative Adversarial Networks (GANs) for hallucinating plausible color and depths in the initially occluded areas. We demonstrate the effectiveness of our approach for novel scene view synthesis from a single image.

Dynamic resolution control in a laser projection-based stereolithography system

PurposeIn a typical additive manufacturing (AM) system, it is critical to make a trade-off between the resolution and the build area for applications in which varied dimensions, feature sizes and accuracies are desired. Conventional solutions to this challenge are based on curing of multiple areas with a single high resolution and stitching them to form a large layer. However, because of the lack of the capability in adjusting resolution dynamically, such stitching approaches will elongate the build time greatly in some cases. To address the challenge without sacrificing the build speed, this paper aims to design and develop a novel AM system with dynamic resolution control capability.Design/methodology/approachA laser projector is adopted in a vat photopolymerization system. The laser projection system has unique properties, including focus-free operation and capability to produce dynamic mask image irrespective of any surface (flat or curved). By translating the projector along the building direction, the pixel size can be adjusted dynamically within a certain range. Consequently, the build area and resolution could be tuned dynamically in the hardware testbed. Besides, a layered depth image (LDI) algorithm is used to construct mask images with varied resolutions. The curing characteristics under various resolution settings are quantified, and accordingly, a process planning approach for fabricating models with dynamically controlled resolutions is developed.FindingsA laser projection-based stereolithography (SL) system could tune resolution dynamically during the building process. Such a dynamic resolution control approach completely addresses the build size-resolution dilemma in vat photopolymerization AM processes without sacrificing the build speed. Through fabricating layers with changing resolutions instead of a single resolution, various build areas and feature sizes could be produced precisely, with optimized build speed.Originality/valueA focus-free laser projector is investigated and adopted in a SL system for the first time. The material curing characteristics with changing focal length and therefore changing light intensities are explored. The related digital mask image planning and process control methods are developed. In digital mask image planning, it is the first attempt to adopt the LDI algorithm, to identify proper resolution settings for fabricating a sliced layer precisely and quickly. In the process of characterizing material curing properties, parametric dependence of curing properties on focal length has been unveiled. This research contributes to the advancement of AM by addressing the historical dilemma of the resolution and build size, and optimizing the build speed meanwhile.

Virtual view synthesis using layered depth image generation and depth-based inpainting for filling disocclusions and translucent disocclusions

View synthesis is an efficient solution to produce content for 3DTV and FTV. However, proper handling of the disocclusions is a major challenge in the view synthesis. Inpainting methods offer solutions for handling disocclusions, though limitations in foreground-background classification causes the holes to be filled with inconsistent textures. Moreover, the state-of-the art methods fail to identify and fill disocclusions in intermediate distances between foreground and background through which background may be visible in the virtual view (translucent disocclusions). Aiming at improved rendering quality, we introduce a layered depth image (LDI) in the original camera view, in which we identify and fill occluded background so that when the LDI data is rendered to a virtual view, no disocclusions appear but views with consistent data are produced also handling translucent disocclusions. Moreover, the proposed foreground-background classification and inpainting fills the disocclusions with neighboring background texture consistently. Based on the objective and subjective evaluations, the proposed method outperforms the state-of-the art methods at the disocclusions.

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

Depth-image-based rendering (DIBR) is a commonly used method for synthesizing additional views using video-plus-depth (V+D) format. A critical issue with DIBR-based view synthesis is the lack of information behind foreground objects. This lack is manifested as disocclusions, holes, next to the foreground objects in rendered virtual views as a consequence of the virtual camera “seeing” behind the foreground object. The disocclusions are larger in the extrapolation case, i.e. the single camera case. Texture synthesis methods (inpainting methods) aim to fill these disocclusions by producing plausible texture content. However, virtual views inevitably exhibit both spatial and temporal inconsistencies at the filled disocclusion areas, depending on the scene content. In this paper, we propose a layered depth image (LDI) approach that improves the spatio-temporal consistency. In the process of LDI generation, depth information is used to classify the foreground and background in order to form a static scene sprite from a set of neighboring frames. Occlusions in the LDI are then identified and filled using inpainting, such that no disocclusions appear when the LDI data is rendered to a virtual view. In addition to the depth information, optical flow is computed to extract the stationary parts of the scene and to classify the occlusions in the inpainting process. Experimental results demonstrate that spatio-temporal inconsistencies are significantly reduced using the proposed method. Furthermore, subjective and objective qualities are improved compared to state-of-the-art reference methods.

Depth of field synthesis from sparse views

Computer generated images are most easily generated as pinhole images whereas images obtained with optical lenses exhibit a Depth-of-Field (DOF) effect. This is due to the fact that optical lenses gather light across finite apertures whereas the simulation of a pinhole lens means that the light is gathered through an infinitesimal small aperture, thus producing sharp images at any depth. Simulating the physical process of gathering light across a finite aperture can be done for example with distributed ray tracing, but it is computationally much more expensive than the simulation through an infinitesimal aperture. The usual way of simulating lens effects is therefore to produce a pinhole image and then post processes the image to approximate the DOF. Post processing algorithms are fast but suffer from incorrect visibilities. In this paper, we propose a novel algorithm that tackles the visibility issue with a sparse set of views rendered through the optical center of the lens and several peripheral viewpoints distributed on the lens. All peripheral images are warped towards the central view to create a Layered-Depth-Image (LDI), so that all observed 3D points located on the same central view-ray are stacked on the same pixel of the LDI. Then, each pixel in the LDI is conceptually scattered into a Point-Spread-Function (PSF) and blended in depth order. While the scatter method is very inefficient on a GPU, we propose a selective-gather method for DOF synthesis, which scans the neighborhood of a pixel and blends the colors from the PSFs covering the pixel. Experiments show that the proposed algorithm can synthesize high-quality DOF effects close to the results of distributed ray tracing but at a much higher speed.

Visual Hull-Based Geometric Data Compression of a 3-D Object

As image-based 3-D modeling is used in a variety of applications, accordingly, the compression of 3-D object geometry represented by multiple images becomes an important task. This paper presents a model-based approach to predict the geometric structure of an object using its visual hull. A visual hull is a geometric entity generated by shape-from-silhouette (SFS), and consequently it largely follows the overall shape of the object. The construction of a visual hull is computationally inexpensive and a visual hull can be encoded with relatively small amount of bits because it can be represented with 2-D silhouette images. Therefore, when it comes to the predictive compression of object’s geometric data, the visual hull should be an effective predictor. In the proposed method, the geometric structure of an object is represented by a layered depth image (LDI), and a visual hull from the LDI data is computed via silhouette generation and SFS. The geometry of an object is predicted with the computed visual hull, and the visual hull data with its prediction errors are encoded. Simulation results show that the proposed predictive coding based on visual hull outperforms the previous image-based methods and the partial surface-based method.

Efficient Depth Propagation for Constructing a Layered Depth Image from a Single Image

AbstractIn this paper, we propose an interactive technique for constructing a 3D scene via sparse user inputs. We represent a 3D scene in the form of a Layered Depth Image (LDI) which is composed of a foreground layer and a background layer, and each layer has a corresponding texture and depth map. Given user‐specified sparse depth inputs, depth maps are computed based on superpixels using interpolation with geodesic‐distance weighting and an optimization framework. This computation is done immediately, which allows the user to edit the LDI interactively. Additionally, our technique automatically estimates depth and texture in occluded regions using the depth discontinuity. In our interface, the user paints strokes on the 3D model directly. The drawn strokes serve as 3D handles with which the user can pull out or push the 3D surface easily and intuitively with real‐time feedback. We show our technique enables efficient modeling of LDI that produce sufficient 3D effects.

계층적 깊이 영상으로 표현된 다시점 비디오에 대한 H.264 부호화 기술

Multi-view video including depth image is necessary to develop a new compression encoding technique for storage and transmission, because of a huge amount of data. Layered depth image is an efficient representation method of multi-view video data. This method makes a data structure that is synthesis of multi-view color and depth image. This efficient method to compress new contents is suggested to use layered depth image representation and to apply for video compression encoding by using 3D warping. This paper proposed enhanced compression method using layered depth image representation and H.264/AVC video coding technology. In experimental results, we confirmed high compression performance and good quality of reconstructed image.

Extended Layered Depth Image Representation in Multiview Navigation

Emerging applications in multiview streaming look for providing interactive navigation services to video players. The user can ask for information from any viewpoint with a minimum transmission delay. The purpose is to provide user with as much information as possible with least number of redundancies. The recent concept of navigation segment representation consists of regrouping a given number of viewpoints in one signal and transmitting them to the users according to their navigation path. The question of the best description strategy of these navigation segments is however still open. In this paper, we propose to represent and code navigation segments by a method that extends the recent layered depth image (LDI) format. It consists of describing the scene from a viewpoint with multiple images organized in layers corresponding to the different levels of occluded objects. The notion of extended LDI comes from the fact that the size of this image is adapted to take into account the sides of the scene also, in contrary to classical LDI. The obtained results show a significant rate-distortion gain compared to classical multiview compression approaches in navigation scenario.

다중 깊이 영상을 이용한 볼륨-표면 혼합 가시화

As volume rendering has been applied for computer game, the visualization of volume data with surface data in one scene has been required. Though a hybrid rendering of volume and surface data have been developed using the GPGPU functionality, computer games which run on low-level hardware are difficult to perform the hybrid rendering. In this paper, we propose a new hybrid rendering based on DirectX 9.0 and general hardware. We generate the layered depth images from surface data using a new method to reduce the depth complexity and generation time. Then, we perform the hybrid rendering using the layered depth images. In the rendering process, we suggest a new method to transform the coordinate system from a surface coordinate to a volume coorinate and propose an accelerated rendering technique. As the result, we can perform volume-surface hybrid rendering in an efficient way.

Conservative sampling of solids in image space.

Conservative sampling samples boundary-representation (B-rep) solid models into layered depth images (LDIs). The resulting models have a closed boundary and are guaranteed to bound the input B-rep models on the rays of LDIs. This approach can be fully implemented by shader programs supported by various graphics hardware. Experimental results demonstrate this approach's efficiency; applications of it to evaluating intersecting volumes and computing Minkowski sums show its versatility.

GPU-based offset surface computation using point samples

We present an efficient algorithm to perform approximate offsetting operations on geometric models using GPUs. Our approach approximates the boundary of an object with point samples and computes the offset by merging the balls centered at these points. The underlying approach uses Layered Depth Images (LDI) to organize the samples into structured points and performs parallel computations using multiple cores. We use spatial hashing to accelerate intersection queries and balance the workload among various cores. Furthermore, the problem of offsetting with a large distance is decomposed into successive offsetting using smaller distances. We derive bounds on the accuracy of offset computation as a function of the sampling rate of LDI and offset distance. In practice, our GPU-based algorithm can accurately compute offsets of models represented using hundreds of thousands of points in a few seconds on a GeForce GTX 580 GPU. We observe more than 100 times speedup over prior serial CPU-based approximate offset computation algorithms.

Parallel and efficient Boolean on polygonal solids

We present a novel framework which can efficiently evaluate approximate Boolean set operations for B-rep models by highly parallel algorithms. This is achieved by taking axis-aligned surfels of Layered Depth Images (LDI) as a bridge and performing Boolean operations on the structured points. As compared with prior surfel-based approaches, this paper has much improvement. Firstly, we adopt key-data pairs to store LDI more compactly. Secondly, robust depth peeling is investigated to overcome the bottleneck of layer-complexity. Thirdly, an out-of-core tiling technique is presented to overcome the limitation of memory. Real-time feedback is provided by streaming the proposed pipeline on the many-core graphics hardware.

修正深度剥離を用いた高速衝突判定手法

We propose a fast collision detection method that uses graphics hardware and includes self-collision and self-collision and deformable objects as its targets. The method uses a layered depth image (LDI), which is generated by depth peeling, and transform feedback. We modify the depth peeling method so that it can peel object surfaces exactly and uses LDI representations for fast collision detection. In addition, transform feedback makes the collision detection process very fast since there is no need to read back from the GPU to the CPU. The proposed method was implemented in a PC having an NVIDIA GeForce 9800 GT graphics card and applied to two objects consisting of 10,000 triangles in total at an image-space resolution of 400 × 400 pixels. As a result, the collision detection time was about 24 ms with 96% detection.

Efficient Depth Compression Based on Partial Surface for 3-D Object Represented by Layered Depth Image

To build a compact and photorealistic representation of a 3-D object without using polygonal mesh, 3-D representations of an object based on images accompanied by depth values were developed. In this letter, we propose a new compression system for the depth component of layered depth image (LDI), one of these representations, which represents a 3-D object. In order to achieve a high compression performance, it is necessary to group more correlated depth data. For this purpose we introduce the concept of partial surfaces from depth point cloud in LDI and derive a depth compression algorithm based on them generated from LDI. Neighboring depth points on the generated partial surface are likely to have high correlation, since it is a curved surface composed of surface points of the object. From experimental results, our proposed compression method outperforms other previous methods in terms of compression efficiency.

Approximate Boolean Operations on Large Polyhedral Solids with Partial Mesh Reconstruction

We present a new approach to compute the approximate Boolean operations of two freeform polygonal mesh solids efficiently with the help of Layered Depth Images (LDIs). After applying the LDI sampling-based membership classification, the most challenging part, a trimmed adaptive contouring algorithm, is developed to reconstruct the mesh surface from the LDI samples near the intersected regions and stitch it to the boundary of the retained surfaces. Our method of approximate Boolean operations holds the advantage of numerical robustness as the approach uses volumetric representation. However, unlike other methods based on volumetric representation, we do not damage the facets in nonintersected regions, thus preserving geometric details much better and speeding up the computation as well. We show that the proposed method can successfully compute the Boolean operations of free-form solids with a massive number of polygons in a few seconds.