Commodity Graphics Hardware Research Articles

Dense tracking, mapping and scene labeling using a depth camera

We present a system for dense tracking, 3D reconstruction, and object detection of desktop-like environments, using a depth camera; the Kinect sensor. The camera is moved by hand meanwhile its pose is estimated, and a dense model, with evolving color information of the scene, is constructed. Alternatively, the user can couple the object detection module (YOLO: you only look once [1]) for detecting and propagating to the model information of categories of objects commonly found over desktops, like monitors, keyboards, books, cups, and laptops, getting a model with color associated to object categories. The camera pose is estimated using a model-to-frame technique with a coarse-to-fine iterative closest point algorithm (ICP), achieving a drift-free trajectory, robustness to fast camera motion and to variable lighting conditions. Simultaneously, the depth maps are fused into the volumetric structure from the estimated camera poses. For visualizing an explicit representation of the scene, the marching cubes algorithm is employed. The tracking, fusion, marching cubes, and object detection processes were implemented using commodity graphics hardware for improving the performance of the system. We achieve outstanding results in camera pose, high quality of the model’s color and geometry, and stability in color from the detection module (robustness to wrong detections) and successful management of multiple instances of the same category.

Evaluation of a BVH Construction Accelerator Architecture for High-Quality Visualization

The ever-increasing demands of computer graphics applications have motivated the evolution of computer graphics hardware over the last 20 years. Early commodity graphics hardware was largely based on fixed-function components offering little flexibility. The gradual replacement of fixed-function hardware with more general-purpose instruction processors, has enabled GPUs to deliver visual experiences more tailored to specific applications. This trend has culminated in modern GPUs essentially being programmable stream processors, capable of supporting a wide variety of applications in and outside of computer graphics. However, the growing concern of power efficiency in modern processors, coupled with an increasing demand for supporting next-generation graphics pipelines, has re-invigorated the debate on the use of fixed-function accelerators in these platforms. In this paper, we conduct a study of a heterogeneous, system-on-chip solution for the construction of a highly important data structure for computer graphics: the bounding volume hierarchy. This design incorporates conventional CPU cores alongside a fixed-function accelerator prototyped on a reconfigurable logic fabric. Our study supports earlier, simulation-only studies which argue for the introduction of this class of accelerator in future graphics processors.

Live intrinsic video

Intrinsic video decomposition refers to the fundamentally ambiguous task of separating a video stream into its constituent layers, in particular reflectance and shading layers. Such a decomposition is the basis for a variety of video manipulation applications, such as realistic recoloring or retexturing of objects. We present a novel variational approach to tackle this underconstrained inverse problem at real-time frame rates, which enables on-line processing of live video footage. The problem of finding the intrinsic decomposition is formulated as a mixed variational ℓ 2 - ℓ p -optimization problem based on an objective function that is specifically tailored for fast optimization. To this end, we propose a novel combination of sophisticated local spatial and global spatio-temporal priors resulting in temporally coherent decompositions at real-time frame rates without the need for explicit correspondence search. We tackle the resulting high-dimensional, non-convex optimization problem via a novel data-parallel iteratively reweighted least squares solver that runs on commodity graphics hardware. Real-time performance is obtained by combining a local-global solution strategy with hierarchical coarse-to-fine optimization. Compelling real-time augmented reality applications, such as recoloring, material editing and retexturing, are demonstrated in a live setup. Our qualitative and quantitative evaluation shows that we obtain high-quality real-time decompositions even for challenging sequences. Our method is able to outperform state-of-the-art approaches in terms of runtime and result quality -- even without user guidance such as scribbles.

Real-time lens distortion correction: speed, accuracy and efficiency

Optical lens systems suffer from nonlinear geometrical distortion. Optical imaging applications such as image-enhanced endoscopy and image-based bronchoscope tracking require correction of this distortion for accurate localization, tracking, registration, and measurement of image features. Real-time capability is desirable for interactive systems and live video. The use of a texture-mapping graphics accelerator, which is standard hardware on current motherboard chipsets and add-in video graphics cards, to perform distortion correction is proposed. Mesh generation for image tessellation, an error analysis, and performance results are presented. It is shown that distortion correction using commodity graphics hardware is substantially faster than using the main processor and can be performed at video frame rates (faster than 30 frames per second), and that the polar-based method of mesh generation proposed here is more accurate than a conventional grid-based approach. Using graphics hardware to perform distortion correction is not only fast and accurate but also efficient as it frees the main processor for other tasks, which is an important issue in some real-time applications.

State‐of‐the‐Art in Compressed GPU‐Based Direct Volume Rendering

AbstractGreat advancements in commodity graphics hardware have favoured graphics processing unit (GPU)‐based volume rendering as the main adopted solution for interactive exploration of rectilinear scalar volumes on commodity platforms. Nevertheless, long data transfer times and GPU memory size limitations are often the main limiting factors, especially for massive, time‐varying or multi‐volume visualization, as well as for networked visualization on the emerging mobile devices. To address this issue, a variety of level‐of‐detail (LOD) data representations and compression techniques have been introduced. In order to improve capabilities and performance over the entire storage, distribution and rendering pipeline, the encoding/decoding process is typically highly asymmetric, and systems should ideally compress at data production time and decompress on demand at rendering time. Compression and LOD pre‐computation does not have to adhere to real‐time constraints and can be performed off‐line for high‐quality results. In contrast, adaptive real‐time rendering from compressed representations requires fast, transient and spatially independent decompression. In this report, we review the existing compressed GPU volume rendering approaches, covering sampling grid layouts, compact representation models, compression techniques, GPU rendering architectures and fast decoding techniques.

Real-time 3D reconstruction at scale using voxel hashing

Online 3D reconstruction is gaining newfound interest due to the availability of real-time consumer depth cameras. The basic problem takes live overlapping depth maps as input and incrementally fuses these into a single 3D model. This is challenging particularly when real-time performance is desired without trading quality or scale. We contribute an online system for large and fine scale volumetric reconstruction based on a memory and speed efficient data structure. Our system uses a simple spatial hashing scheme that compresses space, and allows for real-time access and updates of implicit surface data, without the need for a regular or hierarchical grid data structure. Surface data is only stored densely where measurements are observed. Additionally, data can be streamed efficiently in or out of the hash table, allowing for further scalability during sensor motion. We show interactive reconstructions of a variety of scenes, reconstructing both fine-grained details and large scale environments. We illustrate how all parts of our pipeline from depth map pre-processing, camera pose estimation, depth map fusion, and surface rendering are performed at real-time rates on commodity graphics hardware. We conclude with a comparison to current state-of-the-art online systems, illustrating improved performance and reconstruction quality.

Scalable real-time volumetric surface reconstruction

We address the fundamental challenge of scalability for real-time volumetric surface reconstruction methods. We design a memory efficient, hierarchical data structure for commodity graphics hardware, which supports live reconstruction of large-scale scenes with fine geometric details. Our sparse data structure fuses overlapping depth maps from a moving depth camera into a single volumetric representation, from which detailed surface models are extracted. Our hierarchy losslessly streams data bidirectionally between GPU and host, allowing for unbounded reconstructions. Our pipeline, comprised of depth map post-processing, camera pose estimation, volumetric fusion, surface extraction, and streaming, runs entirely in real-time. We experimentally demonstrate that a shallow hierarchy with relatively large branching factors yields the best memory/speed tradeoff, consuming an order of magnitude less memory than a regular grid. We compare an implementation of our data structure to existing methods and demonstrate higher-quality reconstructions on a variety of large-scale scenes, all captured in real-time.

View‐Dependent Realtime Rendering of Procedural Facades with High Geometric Detail

AbstractWe present an algorithm for realtime rendering of large‐scale city models with procedurally generated facades. By using highly detailed assets like windows, doors, and decoration such city models can provide an extremely high geometric level of detail but on the downside they also consist of billions of polygons which makes it infeasible to even store them as explicit polygonal meshes. Moreover, when rendering urban scenes usually only a very small fraction of the city is actually visible which calls for effective culling mechanisms. For procedural textures there are efficient screen space techniques that evaluate, e.g., a split grammar on a per‐pixel basis in the fragment shader and thus render a textured facade in a view dependent manner. We take this idea further by introducing 3D geometric detail in addition to flat textures. Our approach is a two‐pass procedure that first renders a flat procedural facade. During rasterization the fragment shader triggers the instantiation of a detailed asset whenever a geometric facade element is potentially visible. The set of instantiated detail models are then rendered in a second pass. The major challenges arise from the fact that geometric details belonging to a facade can be visible even if the base polygon of the facade itself is not visible. Hence we propose measures to conservatively estimate visibility without introducing excessive redundancy. We further extend our technique by a simple level of detail mechanism that switches to baked textures (of the assets) depending on the distance to the camera. We demonstrate that our technique achieves realtime frame rates for large‐scale city models with massive detail on current commodity graphics hardware.

Octree-based fusion for realtime 3D reconstruction

This paper proposes an octree-based surface representation for KinectFusion, a realtime reconstruction technique of in-door scenes using a low-cost moving depth camera and a commodity graphics hardware. In KinectFusion, the scene is represented as a signed distance function (SDF) and stored as an uniform grid of voxels. Though the grid-based SDF is suitable for parallel computation in graphics hardware, most of the storage are wasted, because the geometry is very sparse in the scene volume. In order to reduce the memory cost and save the computation time, we represent the SDF in an octree, and developed several octree-based algorithms for reconstruction update and surface prediction that are suitable for parallel computation in graphics hardware. In the reconstruction update step, the octree nodes are adaptively split in breath-first order. To handle scenes with moving objects, the corresponding nodes are automatically detected and removed to avoid storage overflow. In the surface prediction step, an octree-based ray tracing method is adopted and parallelized for graphic hardware. To further reduce the computation time, the octree is organized into four layers, called top layer, branch layer, middle layer and data layer. The experiments showed that, the proposed method consumes only less than 10% memory of original KinectFusion method, and achieves faster performance. Consequently, it can reconstruct scenes with more than 10 times larger size than the original KinectFusion on the same hardware setup.

A GPU Task-Parallel Model with Dependency Resolution

A task-parallel approach to programming commodity graphics hardware is useful for implementing irregular parallel workloads with dependencies, particularly for applications such as video encoding and backtracking algorithms

Adaptive mesh subdivision for efficient light baking

Light baking has long been a popular technique for real-time rendering. It usually precomputes and bakes the global lighting effects as vertex attributes or textures. Vertex baking requires less memory but can cause artifacts for large triangles. Texture baking can avoid this and generate a high-quality visual effect in real-time rendering. However, it requires significant memory consumption, which may limit the real-time performance and usage. To address this problem, we propose an adaptive mesh subdivision algorithm for memory-efficient light baking, including a fast triangle subdivision level determination method and an optimized solution to calculate vertex colors. Only the subdivided mesh is required during the real-time rendering. Therefore, memory requirements can be significantly reduced while keeping the visual effect. Besides, the subdivision level is allowed to be intuitively controlled by users with a specified parameter. Our algorithm can be easily implemented on commodity graphics hardware and integrated in existing real-time applications such as online preview systems.

Graphical processing unit implementation of an integrated shape-based active contour: Application to digital pathology

Commodity graphics hardware has become a cost-effective parallel platform to solve many general computational problems. In medical imaging and more so in digital pathology, segmentation of multiple structures on high-resolution images, is often a complex and computationally expensive task. Shape-based level set segmentation has recently emerged as a natural solution to segmenting overlapping and occluded objects. However the flexibility of the level set method has traditionally resulted in long computation times and therefore might have limited clinical utility. The processing times even for moderately sized images could run into several hours of computation time. Hence there is a clear need to accelerate these segmentations schemes. In this paper, we present a parallel implementation of a computationally heavy segmentation scheme on a graphical processing unit (GPU). The segmentation scheme incorporates level sets with shape priors to segment multiple overlapping nuclei from very large digital pathology images. We report a speedup of 19× compared to multithreaded C and MATLAB-based implementations of the same scheme, albeit with slight reduction in accuracy. Our GPU-based segmentation scheme was rigorously and quantitatively evaluated for the problem of nuclei segmentation and overlap resolution on digitized histopathology images corresponding to breast and prostate biopsy tissue specimens.

Approximate Radiosity Using Stochastic Depth Buffering

Abstract This paper presents a simple technique for computing approximate radiosity by using a novel approach to visibility determination called stochastic depth buffering. By rendering random depth buffer values and using a simple sampling scheme, we gather light from parallel, global directions using the GPU. The technique makes it possible to quickly update indirect light on commodity graphics hardware. It is easy to implement and offers good trade-offs between performance and visual quality.

Parallel fuzzy connected image segmentation on GPU

Image segmentation techniques using fuzzy connectedness (FC) principles have shown their effectiveness in segmenting a variety of objects in several large applications. However, one challenge in these algorithms has been their excessive computational requirements when processing large image datasets. Nowadays, commodity graphics hardware provides a highly parallel computing environment. In this paper, the authors present a parallel fuzzy connected image segmentation algorithm implementation on NVIDIA's compute unified device Architecture (CUDA) platform for segmenting medical image data sets. In the FC algorithm, there are two major computational tasks: (i) computing the fuzzy affinity relations and (ii) computing the fuzzy connectedness relations. These two tasks are implemented as CUDA kernels and executed on GPU. A dramatic improvement in speed for both tasks is achieved as a result. Our experiments based on three data sets of small, medium, and large data size demonstrate the efficiency of the parallel algorithm, which achieves a speed-up factor of 24.4x, 18.1x, and 10.3x, correspondingly, for the three data sets on the NVIDIA Tesla C1060 over the implementation of the algorithm on CPU, and takes 0.25, 0.72, and 15.04 s, correspondingly, for the three data sets. The authors developed a parallel algorithm of the widely used fuzzy connected image segmentation method on the NVIDIA GPUs, which are far more cost- and speed-effective than both cluster of workstations and multiprocessing systems. A near-interactive speed of segmentation has been achieved, even for the large data set.

Parallel mutual information estimation for inferring gene regulatory networks on GPUs

BackgroundMutual information is a measure of similarity between two variables. It has been widely used in various application domains including computational biology, machine learning, statistics, image processing, and financial computing. Previously used simple histogram based mutual information estimators lack the precision in quality compared to kernel based methods. The recently introduced B-spline function based mutual information estimation method is competitive to the kernel based methods in terms of quality but at a lower computational complexity.ResultsWe present a new approach to accelerate the B-spline function based mutual information estimation algorithm with commodity graphics hardware. To derive an efficient mapping onto this type of architecture, we have used the Compute Unified Device Architecture (CUDA) programming model to design and implement a new parallel algorithm. Our implementation, called CUDA-MI, can achieve speedups of up to 82 using double precision on a single GPU compared to a multi-threaded implementation on a quad-core CPU for large microarray datasets. We have used the results obtained by CUDA-MI to infer gene regulatory networks (GRNs) from microarray data. The comparisons to existing methods including ARACNE and TINGe show that CUDA-MI produces GRNs of higher quality in less time.ConclusionsCUDA-MI is publicly available open-source software, written in CUDA and C++ programming languages. It obtains significant speedup over sequential multi-threaded implementation by fully exploiting the compute capability of commonly used CUDA-enabled low-cost GPUs.

Grid-based SAH BVH construction on a GPU

We present an efficient algorithm for building an adaptive bounding volume hierarchy (BVH) in linear time on commodity graphics hardware using CUDA. BVHs are widely used as an acceleration data str...

Non‐Linear Beam Tracing on a GPU

AbstractBeam tracing combines the flexibility of ray tracing and the speed of polygon rasterization. However, beam tracing so far only handles linear transformations; thus, it is only applicable to linear effects such as planar mirror reflections but not to non‐linear effects such as curved mirror reflection, refraction, caustics and shadows. In this paper, we introduce non‐linear beam tracing to render these non‐linear effects. Non‐linear beam tracing is highly challenging because commodity graphics hardware supports only linear vertex transformation and triangle rasterization. We overcome this difficulty by designing a non‐linear graphics pipeline and implementing it on top of a commodity GPU. This allows beams to be non‐linear where rays within the same beam do not have to be parallel or intersect at a single point. Using these non‐linear beams, real‐time GPU applications can render secondary rays via polygon streaming similar to how they render primary rays. A major strength of this methodology is that it naturally supports fully dynamic scenes without the need to pre‐store a scene database. Utilizing our approach, non‐linear ray tracing effects can be rendered in real‐time on a commodity GPU under a unified framework.

GPU-accelerated 3D Bayesian image reconstruction from Compton scattered data

This paper describes the development of fast Bayesian reconstruction methods for Compton cameras using commodity graphics hardware. For fast iterative reconstruction, not only is it important to increase the convergence rate, but also it is equally important to accelerate the computation of time-consuming and repeated operations, such as projection and backprojection. Since the size of the system matrix for a typical Compton camera is intractably large, it is impractical to use a conventional caching scheme that stores the pre-calculated elements of a system matrix and uses them for the calculation of projection and backprojection. In this paper we propose GPU (graphics processing unit)-accelerated methods that can rapidly perform conical projection and backprojection on the fly. Since the conventional ray-based backprojection method is inefficient for parallel computing on GPUs, we develop voxel-based conical backprojection methods using two different approximation schemes. In the first scheme, we approximate the intersecting chord length of the ray passing through a voxel by the perpendicular distance from the center to the ray. In the second scheme, each voxel is regarded as a dimensionless point rather than a cube so that the backprojection can be performed without the need for calculating intersecting chord lengths or their approximations. Our simulation studies show that the GPU-based method dramatically improves the computational speed with only minor loss of accuracy in reconstruction. With the development of high-resolution detectors, the difference in the reconstruction accuracy between the GPU-based method and the CPU-based method will eventually be negligible.

Accelerating urban fast response Lagrangian dispersion simulations using inexpensive graphics processor parallelism

Owing to the potential consequences associated with accidental or deliberate releases of chemical or biological agents in urban areas, fast response urban dispersion models must rapidly provide solutions that can be easily analyzed by researchers and emergency responders. In this paper, we describe a novel application of an existing Lagrangian dispersion modeling system to achieve real-time simulation and visualization of an urban plume that a user can interact with in a virtual environment (VE) through the utilization of commodity graphics hardware, utilizing the highly parallel computational capabilities available on graphics processing units (GPU). GPUs have quickly developed from video game technology to open up new avenues for enhancing simulation performance and visualization of engineering and science applications. For computer graphics applications, GPUs provide highly parallel and inexpensive data paths for processing geometry and pixels, but for simulation these parallel paths are exploited for solving general problems. In this paper, a newly developed dispersion model (GPU Plume) is tested against an analytical solution, a CPU implementation of the Lagrangian dispersion model and wind tunnel data for dispersion around a single building. GPU Plume is shown to provide results that are similar in accuracy to the CPU model, but with computation times that are up to two orders magnitude smaller. In addition, challenges associated with the implementation of Lagrangian dispersion models onto the GPU architecture are discussed in this paper.

Viewpoints: A High-Performance High-Dimensional Exploratory Data Analysis Tool

ABSTRACTScientific data sets continue to increase in both size and complexity. In the past, dedicated graphics systems at supercomputing centers were required to visualize large data sets, but as the price of commodity graphics hardware has dropped and its capability has increased, it is now possible, in principle, to view large complex data sets on a single workstation. To do this in practice, an investigator will need software that is written to take advantage of the relevant graphics hardware. The Viewpoints visualization package described herein is an example of such software. Viewpoints is an interactive tool for exploratory visual analysis of large high-dimensional (multivariate) data. It leverages the capabilities of modern graphics boards (GPUs) to run on a single workstation or laptop. Viewpoints is minimalist: it attempts to do a small set of useful things very well (or at least very quickly) in comparison with similar packages today. Its basic feature set includes linked scatter plots with brushing, dynamic histograms, normalization, and outlier detection/removal. Viewpoints was originally designed for astrophysicists, but it has since been used in a variety of fields that range from astronomy, quantum chemistry, fluid dynamics, machine learning, bioinformatics, and finance to information technology server log mining. In this article, we describe the Viewpoints package and show examples of its usage.