Order-independent Transparency Research Articles

Deep and Fast Approximate Order Independent Transparency

AbstractWe present a machine learning approach for efficiently computing order independent transparency (OIT) by deploying a light weight neural network implemented fully on shaders. Our method is fast, requires a small constant amount of memory (depends only on the screen resolution and not on the number of triangles or transparent layers), is more accurate as compared to previous approximate methods, works for every scene without setup and is portable to all platforms running even with commodity GPUs. Our method requires a rendering pass to extract all features that are subsequently used to predict the overall OIT pixel colour with a pre‐trained neural network. We provide a comparative experimental evaluation and shader source code of all methods for reproduction of the experiments.

Efficient Binocular Rendering of Volumetric Density Fields With Coupled Adaptive Cube-Map Ray Marching for Virtual Reality.

Creating visualizations of multiple volumetric density fields is demanding in virtual reality (VR) applications, which often include divergent volumetric density distributions mixed with geometric models and physics-based simulations. Real-time rendering of such complex environments poses significant challenges for rendering quality and performance. This paper presents a novel scheme for efficient real-time rendering of varying translucent volumetric density fields with global illumination (GI) effects on high-resolution binocular VR displays. Our scheme proposes creative solutions to address three challenges involved in the target problem. Firstly, to tackle the doubled heavy workloads of binocular ray marching, we explore the anti-aliasing principles and more advanced potentials of ray marching on interior cube-map faces, and propose a coupled ray-marching technique that converges to multi-resolution cube maps with interleaved adaptive sampling. Secondly, we devise a fully dynamic ambient GI approximation method that leverages spherical-harmonics (SH) transform information of the phase function to reduce the huge amount of ray sampling required for GI while ensuring fidelity. The method catalyzes spatial ray-marching reuse and adaptive temporal accumulation. Thirdly, we deploy a two-phase ray-tracing algorithm with a tiled k-buffer to achieve fast processing of order-independent transparency (OIT) for multiple volume instances. Consequently, high-quality and high-performance real-time dynamic volume rendering can be achieved under constrained budgets controlled by developers. As our solution supports mixed mesh-volume rendering, the test results prove the practical usefulness of our approach for high-resolution binocular VR rendering on hybrid multi-volumetric and geometric environments.

Validation of real-time inside-out tracking and depth realization technologies for augmented reality-based neuronavigation.

Concomitant with the significant advances in computing technology, the utilization of augmented reality-based navigation in clinical applications is being actively researched. In this light, we developed novel object tracking and depth realization technologies to apply augmented reality-based neuronavigation to brain surgery. We developed real-time inside-out tracking based on visual inertial odometry and a visual inertial simultaneous localization and mapping algorithm. The cube quick response marker and depth data obtained from light detection and ranging sensors are used for continuous tracking. For depth realization, order-independent transparency, clipping, and annotation and measurement functions were developed. In this study, the augmented reality model of a brain tumor patient was applied to its life-size three-dimensional (3D) printed model. Using real-time inside-out tracking, we confirmed that the augmented reality model remained consistent with the 3D printed patient model without flutter, regardless of the movement of the visualization device. The coordination accuracy during real-time inside-out tracking was also validated. The average movement error of the X and Y axes was 0.34 ± 0.21 and 0.04 ± 0.08mm, respectively. Further, the application of order-independent transparency with multilayer alpha blending and filtered alpha compositing improved the perception of overlapping internal brain structures. Clipping, and annotation and measurement functions were also developed to aid depth perception and worked perfectly during real-time coordination. We named this system METAMEDIP navigation. The results validate the efficacy of the real-time inside-out tracking and depth realization technology. With these novel technologies developed for continuous tracking and depth perception in augmented reality environments, we are able to overcome the critical obstacles in the development of clinically applicable augmented reality neuronavigation.

Data-based real-time petrochemical gas diffusion simulation approach on virtual reality

BackgroundPetrochemical products possess a high risk of flammability, explosivity, and toxicity, making petrochemical accidents exceedingly destructive. Therefore, disaster analysis, prediction, and real-time simulations have become important means of controlling and reducing accident hazards. MethodsThis study proposes a complete real-time simulation solution of gas diffusion with coordinate and concentration data, which was mainly aimed at simulating the types of harmful gas leakage and diffusion accidents in the petrochemical industry. The rendering effect was more continuous and accurate through grid homogenization and trilinear interpolation. This study presents a data processing and rendering parallelization process to enhance simulation efficiency. Gas concentration and fragment transparency were combined to synthesize transparent pixels in a scene. To ensure the approximate accuracy of the rendering effect, improve the efficiency of real-time rendering, and meet the requirement of intuitive perception using concentration data, a weighted blended order-independent transparency(OIT) with enhanced alpha weight is presented, which can provide a more intuitive perception of the hierarchical information of concentration data while preserving depth information. This study compares and analyzes three OIT algorithms—depth peeling, weighted blended OIT, and weighted blended OIT with enhanced alpha weight—in terms of rendering image quality, rendering time, required memory, and hierarchical information. ResultsUsing weighted blended OIT with an enhanced alpha weight technique, the rendering time was shortened by 53.2% compared with that of the depth peeling algorithm, and the texture memory required was significantly smaller than that of the depth peeling algorithm. The rendering results of weighted blended OIT with an enhanced alpha weight were approximately accurate compared with those of the depth peeling algorithm as the ground truth, and there was no popping when surfaces passed through one another. Simultaneously, compared with weighted blended OIT, weighted blended OIT with an enhanced alpha weight achieved an intuitive perception of the hierarchical information of concentration data.

Towards 3D Scene Understanding Using Differentiable Rendering

Deep learning methods have achieved significant results in many 2D computer vision tasks. To realize similar results in 3D tasks, equipping deep learning pipelines with components that incorporate knowledge about 2D image generation from the 3D scene description is a promising research direction. Rasterization, the standard formulation of the image generation process is not differentiable, and thus not compatible with the deep learning models trained using gradient-based optimization schemes. In recent years, many new approximate differentiable renderers have been proposed to enable compatibility between deep learning methods and image rendering techniques. Differentiable renderers fit naturally into the render-and-compare framework where the 3D scene parameters are estimated iteratively by minimizing the error between the observed image and the image rendered according to the current scene parameter estimate. In this article, we present StilllebenDR, a light-weight, scalable differentiable renderer built as an extension to the openly available Stillleben library. We demonstrate the usability of the proposed differentiable renderer for the task of iterative 3D deformable registration using a latent shape-space model and occluded object pose refinement using order-independent transparency based on analytical gradients and learned scene aggregation.

Wavelet-Based Order Independent Transparency

Rendering transparent objects in real-time is an important factor in improving the realism of the game screen. In order to improve the speed of rendering transparent objects while maintaining the quality, an order-independent transparency (OIT) rendering algorithm is proposed which uses wavelet series to reconstruct the transmittance of transparent objects. Firstly, all transparent fragments corresponding to each pixel are obtained by rasterization when rendering transparent objects. Then, according to the depth value, the absorption rate of the transparent fragments is projected to the wavelet scaling function, and the coefficients are calculated and accumulated to get the information needed to reconstruct the transmittance function per pixel. Finally, the wavelet coefficients saved at the pixel are taken out to reconstruct the transmittance function, and the transmittance is calculated by substituting the depth value to the reconstructed function. And the color of the fragment is accumulated to the final pixel according to the transmittance. The experiment uses transparent Utah teapot, Stanford dragon and other data to test on the GTX 980 graphics card. The results show that compared with the moment-based OIT algorithm, our algorithm has an efficiency improvement of about 10% to 40% in different scenes. Taking the OIT algorithm based on the concurrent linked list as ground truth, the root mean square error (RMSE) of the rendering results in this paper are no more than 1.5%.

Fast raycasting using a compound deep image for virtual point light range determination

The concept of using multiple deep images, under a variety of different names, has been explored as a possible acceleration approach for finding ray-geometry intersections. We leverage recent advances in deep image processing from order independent transparency for fast building of a compound deep image (CDI) using a coherent memory format well suited for raycasting. We explore the use of a CDI and raycasting for the problem of determining distance between virtual point lights (VPLs) and geometry for indirect lighting, with the key raycasting step being a small fraction of total frametime.

Perception enhancement using importance-driven hybrid rendering for augmented reality based endoscopic surgical navigation.

Misleading depth perception may greatly affect the correct identification of complex structures in image-guided surgery. In this study, we propose a novel importance-driven hybrid rendering method to enhance perception for navigated endoscopic surgery. First, the volume structures are enhanced using gradient-based shading to reduce the color information in low-priority regions and improve the distinctions between complicated structures. Second, an importance sorting method based on the order-independent transparency rendering is introduced to intensify the perception of multiple surfaces. Third, volume data are adaptively truncated and emphasized with respect to the perspective orientation and the illustration of critical information for viewing range extension. Various experimental results prove that with the combination of volume and surface rendering, our method can effectively improve the depth distinction of multiple objects both in simulated and clinical scenes. Our importance-driven surface rendering method demonstrates improved average performance and statistical significance as rated by 15 participants (five clinicians and ten non-clinicians) on a five-point Likert scale. Further, the average frame rate of hybrid rendering with thin-layer sectioning reaches 42 fps. Given that the process of the hybrid rendering is fully automatic, it can be utilized in real-time surgical navigation to improve the rendering efficiency and information validity.

Moment-Based Order-Independent Transparency

Compositing transparent surfaces rendered in an arbitrary order requires techniques for order-independent transparency. Each surface color needs to be multiplied by the appropriate transmittance to the eye to incorporate occlusion. Building upon moment shadow mapping, we present a moment-based method for compact storage and fast reconstruction of this depth-dependent function per pixel. We work with the logarithm of the transmittance such that the function may be accumulated additively rather than multiplicatively. Then an additive rendering pass for all transparent surfaces yields moments. Moment-based reconstruction algorithms provide approximations to the original function, which are used for compositing in a second additive pass. We utilize existing algorithms with four or six power moments and develop new algorithms using eight power moments or up to four trigonometric moments. The resulting techniques are completely order-independent, work well for participating media as well as transparent surfaces and come in many variants providing different tradeoffs. We also utilize the same approach for the closely related problem of computing shadows for transparent surfaces.

Hybrid fur rendering: combining volumetric fur with explicit hair strands

Hair is typically modeled and rendered using either explicitly defined hair strand geometry or a volume texture of hair densities. Taken each on their own, these two hair representations have difficulties in the case of animal fur as it consists of very dense and thin undercoat hairs in combination with coarse guard hairs. Explicit hair strand geometry is not well-suited for the undercoat hairs, while volume textures are not well-suited for the guard hairs. To efficiently model and render both guard hairs and undercoat hairs, we present a hybrid technique that combines rasterization of explicitly defined guard hairs with ray marching of a prismatic shell volume with dynamic resolution. The latter is the key to practical combination of the two techniques, and it also enables a high degree of detail in the undercoat. We demonstrate that our hybrid technique creates a more detailed and soft fur appearance as compared with renderings that only use explicitly defined hair strands. Finally, our rasterization approach is based on order-independent transparency and renders high-quality fur images in seconds.

VoxLink—Combining sparse volumetric data and geometry for efficient rendering

Processing and visualizing large scale volumetric and geometric datasets is mission critical in an increasing number of applications in academic research as well as in commercial enterprise. Often the datasets are, or can be processed to become, sparse. In this paper, we present VoxLink, a novel approach to render sparse volume data in a memory-efficient manner enabling interactive rendering on common, offthe- shelf graphics hardware. Our approach utilizes current GPU architectures for voxelizing, storing, and visualizing such datasets. It is based on the idea of perpixel linked lists (ppLL), an A-buffer implementation for order-independent transparency rendering. The method supports voxelization and rendering of dense semi-transparent geometry, sparse volume data, and implicit surface representations with a unified data structure. The proposed data structure also enables efficient simulation of global lighting effects such as reflection, refraction, and shadow ray evaluation.

Order-Independent Transparency for Programmable Deferred Shading Pipelines

In this paper, we present a flexible and efficient approach for the integration of order-independent transparency into a deferred shading pipeline. The intermediate buffers for storing fragments to be shaded are extended with a dynamic and memory-efficient storage for transparent fragments. The transparency of an object is not fixed and remains programmable until fragment processing, which allows for the implementation of advanced materials effects, interaction techniques or adaptive fade-outs. Traversing costs for shading the transparent fragments are greatly reduced by introducing a tile-based light-culling pass. During deferred shading, opaque and transparent fragments are shaded and composited in front-to-back order using the retrieved lighting information and a physically-based shading model. In addition, we discuss various configurations of the system and further enhancements. Our results show that the system performs at interactive frame rates even for complex scenarios.

Interactive Isogeometric Volume Visualization with Pixel-Accurate Geometry.

A recent development, called isogeometric analysis, provides a unified approach for design, analysis and optimization of functional products in industry. Traditional volume rendering methods for inspecting the results from the numerical simulations cannot be applied directly to isogeometric models. We present a novel approach for interactive visualization of isogeometric analysis results, ensuring correct, i.e., pixel-accurate geometry of the volume including its bounding surfaces. The entire OpenGL pipeline is used in a multi-stage algorithm leveraging techniques from surface rendering, order-independent transparency, as well as theory and numerical methods for ordinary differential equations. We showcase the efficiency of our approach on different models relevant to industry, ranging from quality inspection of the parametrization of the geometry, to stress analysis in linear elasticity, to visualization of computational fluid dynamics results.

Interactive image filtering for level-of-abstraction texturing of virtual 3D scenes

Texture mapping is a key technology in computer graphics. For the visual design of 3D scenes, in particular, effective texturing depends significantly on how important contents are expressed, e.g., by preserving global salient structures, and how their depiction is cognitively processed by the user in an application context. Edge-preserving image filtering is one key approach to address these concerns. Much research has focused on applying image filters in a post-process stage to generate artistically stylized depictions. However, these approaches generally do not preserve depth cues, which are important for the perception of 3D visualization (e.g., texture gradient). To this end, filtering is required that processes texture data coherently with respect to linear perspective and spatial relationships. In this work, we present an approach for texturing 3D scenes with perspective coherence by arbitrary image filters. We propose decoupled deferred texturing with (1) caching strategies to interactively perform image filtering prior to texture mapping and (2) for each mipmap level separately to enable a progressive level of abstraction, using (3) direct interaction interfaces to parameterize the visualization according to spatial, semantic, and thematic data. We demonstrate the potentials of our method by several applications using touch or natural language inputs to serve the different interests of users in specific information, including illustrative visualization, focus+context visualization, geometric detail removal, and semantic depth of field. The approach supports frame-to-frame coherence, order-independent transparency, multitexturing, and content-based filtering. In addition, it seamlessly integrates into real-time rendering pipelines and is extensible for custom interaction techniques.

Fast sorting for exact OIT of complex scenes

Exact order-independent transparency (OIT) techniques capture all fragments during rasterization. The fragments are then sorted per-pixel by depth and composited in order using alpha transparency. The sorting stage is a bottleneck for high depth complexity scenes, taking 70---95 % of the total time for those investigated. In this paper, we show that typical shader-based sorting speed is impacted by local memory latency and occupancy. We present and discuss the use of both registers and an external merge sort in register-based block sort to better use the memory hierarchy of the GPU for improved OIT rendering performance. This approach builds upon backwards memory allocation, achieving an OIT rendering speed up to 1.7 $$\times $$ × that of the best previous method and 6.3 $$\times $$ × that of the common straight forward OIT implementation. In some cases, the sorting stage is reduced to no longer be the dominant OIT component.

Memory-optimized order-independent transparency with Dynamic Fragment Buffer

Exact order-independent transparency (OIT) rendering is memory demanding because it requires per-pixel blending of an unknown number of fragments that need to be stored and sorted before compositing. In this paper, we describe the Dynamic Fragment Buffer (DFB) algorithm, which efficiently manages memory to perform correct compositing for pixels with varying numbers of fragments. We present a collection of experiments that illustrate the advantages of the DFB algorithm with respect to other OIT algorithms, and analyze the impact of the proposed variations.

Depth-presorted triangle lists

We present a novel approach for real-time rendering of static 3D models front-to-back or back-to-front relative to any viewpoint outside its bounding volume. The approach renders depth-sorted triangles using a single draw-call. At run-time, we replace the traditional sorting strategy of existing algorithms with a faster triangle selection strategy. The selection process operates on an extended sequence of triangles annotated by test planes, created by our off-line preprocessing stage. Based on these test planes, a simple run-time procedure uses the given viewpoint to select a subsequence of triangles for rasterization. Selected subsequences are statically presorted by depth and contain each input triangle exactly once. Our method runs on legacy hardware and renders depth-sorted static models significantly faster than previous approaches. We conclude demonstrating the real-time rendering of order-independent transparency effects.

Evaluation of per-pixel linked lists for distributed rendering and comparative analysis

Rendering scenes with overlapping transparent faces is challenging for several reasons, including sorting geometry, managing multiple per-pixel fragments, and compositing. These questions have been solved for local rendering previously, for example by order-independent transparency approaches like Depth Peeling or A-buffer implementations. We discuss order-independent transparency algorithms in context of distributed rendering for large data sets or simulations which by design split the work load among a number of computation nodes. Furthermore, we evaluate a per-pixel linked list approach for distributed mesh rendering and multi-variant analysis of molecule data sets.

Guest Editor's Introduction: Special Section on the Symposium on Interactive 3D Graphics and Games (I3D)

THIS IEEE Transactions on Visualization and Computer Graphics (TVCG) special section brings to you extended versions of four excellent articles originally presented at the Symposium on Interactive 3D Graphics and Games (I3D) in 2010. Since 1986, I3D has been a venue for cutting-edge computer science research in interactive graphics and human interaction. The field has grown significantly since the founding of I3D, but the symposium has maintained its premier status. Today, I3D is known for the close integration of academia and industry, and such is clear from the mixture of attendees and from the mixture of authors of the articles in this section. The symposium itself is also very popular, for it provides a great venue for meeting fellow researchers, forming new collaborations, and brainstorming new solutions. I3D 2010 received 71 submissions from 19 countries in four continents. Each paper received at least three independent reviews by a group of 86 experts in the field of computer graphics and interactive techniques. After a discussion phase, 23 papers were selected for publication and presentation at the symposium (an acceptance rate of 32 percent). There have been no limits imposed on the number of accepted papers, and the decisions were based purely on the submissions’ merits. Following a tradition started with I3D 2009, we invited the authors of four of the accepted papers to expand their work with new insights and results for publication in this special section of TVCG on I3D. The choice of the invited papers was based on the evaluation and comments of our expert reviewers. These papers went through a complete journal review process, which included multiple iterations of editing and review. “Stochastic Transparency” by Eric Enderton, Erik Sintorn, Peter Shirley, and David Luebke addresses the problem of achieving order-independent transparency for interactive rendering. The solution provided by the authors presents several desirable features: the required memory does not depend on the scene’s depth complexity; it renders a fixed number of passes, with its running time growing linearly with the number of fragments; it is general in the sense that it can replace multiple transparency algorithms for different scenarios; it is also easy to implement, and is a good fit to modern, massively parallel GPUs. The technique provides a practical solution for interactive applications, being capable of rendering all kinds of transparent geometry, and providing a unifying approach for orderindependent transparency, anti-aliasing, and shadowing. “Efficient Sparse Voxel Octrees” by Samuli Laine and Tero Karras explores the use of voxel representations for rendering complex detailed geometry on current and future GPUs. In order to efficiently ray cast the resulting models, the authors propose a compact data structure for storing voxels. In this representation, voxels are augmented with contour information to increase geometry resolution. The paper also describes a new normal compression format for storing high-precision object-space normals. Using the proposed infrastructure, the authors challenge the current graphics trend of using a base mesh augmented with fine details represented by textures and displacement maps: as the amount of color and geometric detail information grows beyond a certain limit, wouldn’t it make more sense to use a voxel representation to store both geometry and its associated attributes? “Frankenrigs: Building Character Rigs from Multiple Sources” by Christian Miller, Okan Arikan, and Don Fussel addresses an important practical problem in 3D character animation. Rigging and skinning a character is a fundamental animation task, but a time-consuming and tedious one even for experts. The authors present an automatic solution for generating good quality rigging and skinning that can greatly simplify the work of animators. Given a character’s 3D mesh with annotated joints, the technique finds good matches for individual parts of the model by scanning a database of partial rigs. By transferring information from the partial rigs to the target mesh, the approach produces a skeleton and skinning weights whose quality are similar to the ones produced by expert animators. “Improving Shape Depiction under Arbitrary Rendering” by Romain Vergne, Romain Pacanowski, Pascal Barla, Xavier Granier, and Christophe Schlick introduces a technique for enhancing shape depiction by scaling the reflected light as a function of curvature and material properties. Their solution can be used with any kind of material and works with both direct and global illumination. It can also be used with different lighting environments, supports inter-reflections, and works in real time. By providing intuitive control over the resulting shading, this 1034 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 17, NO. 8, AUGUST 2011

Stochastic Transparency

Stochastic transparency provides a unified approach to order-independent transparency, antialiasing, and deep shadow maps. It augments screen-door transparency using a random sub-pixel stipple pattern, where each fragment of transparent geometry covers a random subset of pixel samples of size proportional to alpha. This results in correct alpha-blended colors on average, in a single render pass with fixed memory size and no sorting, but introduces noise. We reduce this noise by an alpha correction pass, and by an accumulation pass that uses a stochastic shadow map from the camera. At the pixel level, the algorithm does not branch and contains no read-modify-write loops, other than traditional z-buffer blend operations. This makes it an excellent match for modern massively parallel GPU hardware. Stochastic transparency is very simple to implement and supports all types of transparent geometry, able without coding for special cases to mix hair, smoke, foliage, windows, and transparent cloth in a single scene.