Standard Graphics Pipeline Research Articles

Content-aware texture deformation with dynamic control

Textures improve the appearance of virtual scenes by mapping visual details on the surface of 3D objects. Various scenarios – such as real-time animation, interactive texture modelling, or offline post-production – require textures to be deformed in a controllable and plausible manner. We propose a novel approach to model and control texture deformations, which is easy to implement in a standard graphics pipeline. The deformation is implemented at pixel resolution as a warping in the parametric domain. The warping is controlled locally and dynamically by real-time integration along the streamlines of a pre-computed flow field. We propose a technique to pre-compute the flow field from a simple scalar map representing heterogeneous dynamic behaviors. Moreover, to manage sampling issues arising in over-stretched areas during deformation, we provide a mechanism based on re-sampling and texture synthesis. Warping may alternatively be controlled by deformation of the underlying surface, environment parameters or interactive editing, which demonstrates the versatility of our approach.

Deferred neural rendering

The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering , a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures , which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.

Holographic near-eye displays for virtual and augmented reality

We present novel designs for virtual and augmented reality near-eye displays based on phase-only holographic projection. Our approach is built on the principles of Fresnel holography and double phase amplitude encoding with additional hardware, phase correction factors, and spatial light modulator encodings to achieve full color, high contrast and low noise holograms with high resolution and true per-pixel focal control. We provide a GPU-accelerated implementation of all holographic computation that integrates with the standard graphics pipeline and enables real-time (≥90 Hz) calculation directly or through eye tracked approximations. A unified focus, aberration correction, and vision correction model, along with a user calibration process, accounts for any optical defects between the light source and retina. We use this optical correction ability not only to fix minor aberrations but to enable truly compact, eyeglasses-like displays with wide fields of view (80°) that would be inaccessible through conventional means. All functionality is evaluated across a series of hardware prototypes; we discuss remaining challenges to incorporate all features into a single device.

Computer graphics: principles and practice

1. Introduction. Image Processing as Picture Analysis. The Advantages of Interactive Graphics. Representative Uses of Computer Graphics. Classification of Applications. Development of Hardware and Software for Computer Graphics. Conceptual Framework for Interactive Graphics. 2. Programming in the Simple Raster Graphics Package (SRGP)/. Drawing with SRGP/. Basic Interaction Handling/. Raster Graphics Features/. Limitations of SRGP/. 3. Basic Raster Graphics Algorithms for Drawing 2d Primitives. Overview. Scan Converting Lines. Scan Converting Circles. Scan Convertiing Ellipses. Filling Rectangles. Fillign Polygons. Filling Ellipse Arcs. Pattern Filling. Thick Primiives. Line Style and Pen Style. Clipping in a Raster World. Clipping Lines. Clipping Circles and Ellipses. Clipping Polygons. Generating Characters. SRGP_copyPixel. Antialiasing. 4. Graphics Hardware. Hardcopy Technologies. Display Technologies. Raster-Scan Display Systems. The Video Controller. Random-Scan Display Processor. Input Devices for Operator Interaction. Image Scanners. 5. Geometrical Transformations. 2D Transformations. Homogeneous Coordinates and Matrix Representation of 2D Transformations. Composition of 2D Transformations. The Window-to-Viewport Transformation. Efficiency. Matrix Representation of 3D Transformations. Composition of 3D Transformations. Transformations as a Change in Coordinate System. 6. Viewing in 3D. Projections. Specifying an Arbitrary 3D View. Examples of 3D Viewing. The Mathematics of Planar Geometric Projections. Implementing Planar Geometric Projections. Coordinate Systems. 7. Object Hierarchy and Simple PHIGS (SPHIGS). Geometric Modeling. Characteristics of Retained-Mode Graphics Packages. Defining and Displaying Structures. Modeling Transformations. Hierarchical Structure Networks. Matrix Composition in Display Traversal. Appearance-Attribute Handling in Hierarchy. Screen Updating and Rendering Modes. Structure Network Editing for Dynamic Effects. Interaction. Additional Output Features. Implementation Issues. Optimizing Display of Hierarchical Models. Limitations of Hierarchical Modeling in PHIGS. Alternative Forms of Hierarchical Modeling. 8. Input Devices, Interaction Techniques, and Interaction Tasks. Interaction Hardware. Basic Interaction Tasks. Composite Interaction Tasks. 9. Dialogue Design. The Form and Content of User-Computer Dialogues. User-Interfaces Styles. Important Design Considerations. Modes and Syntax. Visual Design. The Design Methodology. 10. User Interface Software. Basic Interaction-Handling Models. Windows-Management Systems. Output Handling in Window Systems. Input Handling in Window Systems. Interaction-Technique Toolkits. User-Interface Management Systems. 11. Representing Curves and Surfaces. Polygon Meshes. Parametric Cubic Curves. Parametric Bicubic Surfaces. Quadric Surfaces. 12. Solid Modeling. Representing Solids. Regularized Boolean Set Operations. Primitive Instancing. Sweep Representations. Boundary Representations. Spatial-Partitioning Representations. Constructive Solid Geometry. Comparison of Representations. User Interfaces for Solid Modeling. 13. Achromatic and Colored Light. Achromatic Light. Chromatic Color. Color Models for Raster Graphics. Reproducing Color. Using Color in Computer Graphics. 14. The Quest for Visual Realism. Why Realism? Fundamental Difficulties. Rendering Techniques for Line Drawings. Rendering Techniques for Shaded Images. Improved Object Models. Dynamics. Stereopsis. Improved Displays. Interacting with Our Other Senses. Aliasing and Antialiasing. 15. Visible-Surface Determination. Functions of Two Variables. Techniques for Efficient Visible-Surface Determination. Algorithms for Visible-Line Determination. The z-Buffer Algorithm. List-Priority Algorithms. Scan-Line Algorithms. Area-Subdivision Algorithms. Algorithms for Octrees. Algorithms for Curved Surfaces. Visible-Surface Ray Tracing. 16. Illumination And Shading. Illumination Modeling. Shading Models for Polygons. Surface Detail. Shadows. Transparency. Interobject Reflections. Physically Based Illumination Models. Extended Light Sources. Spectral Sampling. Improving the Camera Model. Global Illumination Algorithms. Recursive Ray Tracing. Radiosity Methods. The Rendering Pipeline. 17. Image Manipulation and Storage. What Is an Image? Filtering. Image Processing. Geometric Transformations of Images. Multipass Transformations. Image Compositing. Mechanisms for Image Storage. Special Effects with Images. Summary. 18. Advanced Raster Graphic Architecture. Simple Raster-Display System. Display-Processor Systems. Standard Graphics Pipeline. Introduction to Multiprocessing. Pipeline Front-End Architecture. Parallel Front-End Architectures. Multiprocessor Rasterization Architectures. Image-Parallel Rasterization. Object-Parallel Rasterization. Hybrid-Parallel Rasterization. Enhanced Display Capabilities. 19. Advanced Geometric and Raster Algorithms. Clipping. Scan-Converting Primitives. Antialiasing. The Special Problems of Text. Filling Algorithms. Making copyPixel Fast. The Shape Data Structure and Shape Algebra. Managing Windows with bitBlt. Page Description Languages. 20. Advanced Modeling Techniques. Extensions of Previous Techniques. Procedural Models. Fractal Models. Grammar-Based Models. Particle Systems. Volume Rendering. Physically Based Modeling. Special Models for Natural and Synthetic Objects. Automating Object Placement. 21. Animation. Conventional and Computer-Assisted Animation. Animation Languages. Methods of Controlling Animation. Basic Rules of Animation. Problems Peculiar to Animation. Appendix: Mathematics for Computer Graphics. Vector Spaces and Affine Spaces. Some Standard Constructions in Vector Spaces. Dot Products and Distances. Matrices. Linear and Affine Transformations. Eigenvalues and Eigenvectors. Newton-Raphson Iteration for Root Finding. Bibliography. Index. 0201848406T04062001

Real-time Image Generation for Compressive Light Field Displays

With the invention of integral imaging and parallax barriers in the beginning of the 20th century, glasses-free 3D displays have become feasible. Only today—more than a century later—glasses-free 3D displays are finally emerging in the consumer market. The technologies being employed in current-generation devices, however, are fundamentally the same as what was invented 100 years ago. With rapid advances in optical fabrication, digital processing power, and computational perception, a new generation of display technology is emerging: compressive displays exploring the co-design of optical elements and computational processing while taking particular characteristics of the human visual system into account. In this paper, we discuss real-time implementation strategies for emerging compressive light field displays. We consider displays composed of multiple stacked layers of light-attenuating or polarization-rotating layers, such as LCDs. The involved image generation requires iterative tomographic image synthesis. We demonstrate that, for the case of light field display, computed tomographic light field synthesis maps well to operations included in the standard graphics pipeline, facilitating efficient GPU-based implementations with real-time framerates.

Sense and sidedness in the graphics pipeline via a passage through a separable space

Computer graphics is ostensibly based on projective geometry. The graphics pipeline—the sequence of functions applied to 3D geometric primitives to determine a 2D image—is described in the graphics literature as taking the primitives from Euclidean to projective space, and then back to Euclidean space. This is a weak foundation for computer graphics. An instructor is at a loss: one day entering the classroom and invoking the established and venerable theory of projective geometry while asserting that projective spaces are not separable, and then entering the classroom the following week to tell the students that the standard graphics pipeline performs clipping not in Euclidean, but in projective space—precisely the operation (deciding sidedness, which depends on separability) that was deemed nonsensical. But there is no need to present Blinn and Newell’s algorithm (Comput. Graph. 12, 245–251, 1978; Commun. ACM 17, 32–42, 1974)—the crucial clipping step in the graphics pipeline and, perhaps, the most original knowledge a student learns in a fourth-year computer graphics class—as a clever trick that just works. Jorge Stolfi described in 1991 oriented projective geometry. By declaring the two vectors $(x,y,z,w)^{{\ensuremath {\mathsf {T}}}}$and $(-x,-y,-z,-w)^{{\ensuremath {\mathsf {T}}}}$distinct, Blinn and Newell were already unknowingly working in oriented projective space. This paper presents the graphics pipeline on this stronger foundation.

Deriving linear transformations in three dimensions

This article presents a general technique for solving elementary algebraic problems that arise in 3D geometry and then applies this method to derive explicit formulas for many of the linear transformations used in the standard 3D graphics pipeline.