High-quality Renderings Research Articles

MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned From Image Pairs

Video generation has achieved rapid progress benefiting from high-quality renderings provided by powerful image generators. We regard the video synthesis task as generating a sequence of images sharing the same contents but varying in motions. However, most previous video synthesis frameworks based on pre-trained image generators treat content and motion generation separately, leading to unrealistic generated videos. Therefore, we design a novel framework to build the motion space, aiming to achieve content consistency and fast convergence for video generation. We present MotionVideoGAN, a novel video generator synthesizing videos based on the motion space learned by pre-trained image pair generators. Firstly, we propose an image pair generator named MotionStyleGAN to generate image pairs sharing the same contents and producing various motions. Then we manage to acquire motion codes to edit one image in the generated image pairs and keep the other unchanged. The motion codes help us edit images within the motion space since the edited image shares the same contents with the other unchanged one in image pairs. Finally, we introduce a latent code generator to produce latent code sequences using motion codes for video generation. Our approach achieves state-of-the-art performance on the most complex video dataset ever used for unconditional video generation evaluation, UCF101.

Neural Point Catacaustics for Novel-View Synthesis of Reflections

View-dependent effects such as reflections pose a substantial challenge for image-based and neural rendering algorithms. Above all, curved reflectors are particularly hard, as they lead to highly non-linear reflection flows as the camera moves. We introduce a new point-based representation to compute Neural Point Catacaustics allowing novel-view synthesis of scenes with curved reflectors, from a set of casually-captured input photos. At the core of our method is a neural warp field that models catacaustic trajectories of reflections, so complex specular effects can be rendered using efficient point splatting in conjunction with a neural renderer. One of our key contributions is the explicit representation of reflections with a reflection point cloud which is displaced by the neural warp field, and a primary point cloud which is optimized to represent the rest of the scene. After a short manual annotation step, our approach allows interactive high-quality renderings of novel views with accurate reflection flow. Additionally, the explicit representation of reflection flow supports several forms of scene manipulation in captured scenes, such as reflection editing, cloning of specular objects, reflection tracking across views, and comfortable stereo viewing. We provide the source code and other supplemental material on https://repo-sam.inria.fr/fungraph/neural_catacaustics/

A shared space for families in a children’s hospital.

This article presents a research and design project undertaken to develop a shared space for families and healthcare professionals in a children’s hospital. For space development, the design team conceived the Minimum Viable Prototype Lab, a participatory design workshop. The use of minimum resources was the backbone of the workshop, as participants would focus on functional or atmospheric requirements to design an Acoustic Cabin and a Proximity Room. The use of cardboard sets helped the users focus on functional requirements. High-quality renderings and floor plans were used to generate discussion about space atmospheres. We share the resulting prototypes and the benefits for getting different points of view. We also discuss the role of images, drawings, and renderings to inform the perception and possibilities of space.

Rendering with style

For several decades, researchers have been advancing techniques for creating and rendering 3D digital faces, where a lot of the effort has gone into geometry and appearance capture, modeling and rendering techniques. This body of research work has largely focused on facial skin, with much less attention devoted to peripheral components like hair, eyes and the interior of the mouth. As a result, even with the best technology for facial capture and rendering, in most high-end productions a lot of artist time is still spent modeling the missing components and fine-tuning the rendering parameters to combine everything into photo-real digital renders. In this work we propose to combine incomplete, high-quality renderings showing only facial skin with recent methods for neural rendering of faces, in order to automatically and seamlessly create photo-realistic full-head portrait renders from captured data without the need for artist intervention. Our method begins with traditional face rendering, where the skin is rendered with the desired appearance, expression, viewpoint, and illumination. These skin renders are then projected into the latent space of a pre-trained neural network that can generate arbitrary photo-real face images (StyleGAN2). The result is a sequence of realistic face images that match the identity and appearance of the 3D character at the skin level, but is completed naturally with synthesized hair, eyes, inner mouth and surroundings. Notably, we present the first method for multi-frame consistent projection into this latent space, allowing photo-realistic rendering and preservation of the identity of the digital human over an animated performance sequence, which can depict different expressions, lighting conditions and viewpoints. Our method can be used in new face rendering pipelines and, importantly, in other deep learning applications that require large amounts of realistic training data with ground-truth 3D geometry, appearance maps, lighting, and viewpoint.

Learning ultrasound rendering from cross-sectional model slices for simulated training

PurposeGiven the high level of expertise required for navigation and interpretation of ultrasound images, computational simulations can facilitate the training of such skills in virtual reality. With ray-tracing based simulations, realistic ultrasound images can be generated. However, due to computational constraints for interactivity, image quality typically needs to be compromised.MethodsWe propose herein to bypass any rendering and simulation process at interactive time, by conducting such simulations during a non-time-critical offline stage and then learning image translation from cross-sectional model slices to such simulated frames. We use a generative adversarial framework with a dedicated generator architecture and input feeding scheme, which both substantially improve image quality without increase in network parameters. Integral attenuation maps derived from cross-sectional model slices, texture-friendly strided convolutions, providing stochastic noise and input maps to intermediate layers in order to preserve locality are all shown herein to greatly facilitate such translation task.ResultsGiven several quality metrics, the proposed method with only tissue maps as input is shown to provide comparable or superior results to a state-of-the-art that uses additional images of low-quality ultrasound renderings. An extensive ablation study shows the need and benefits from the individual contributions utilized in this work, based on qualitative examples and quantitative ultrasound similarity metrics. To that end, a local histogram statistics based error metric is proposed and demonstrated for visualization of local dissimilarities between ultrasound images.ConclusionA deep-learning based direct transformation from interactive tissue slices to likeness of high quality renderings allow to obviate any complex rendering process in real-time, which could enable extremely realistic ultrasound simulations on consumer-hardware by moving the time-intensive processes to a one-time, offline, preprocessing data preparation stage that can be performed on dedicated high-end hardware.

Layered neural rendering for retiming people in video

We present a method for retiming people in an ordinary, natural video --- manipulating and editing the time in which different motions of individuals in the video occur. We can temporally align different motions, change the speed of certain actions (speeding up/slowing down, or entirely "freezing" people), or "erase" selected people from the video altogether. We achieve these effects computationally via a dedicated learning-based layered video representation, where each frame in the video is decomposed into separate RGBA layers, representing the appearance of different people in the video. A key property of our model is that it not only disentangles the direct motions of each person in the input video, but also correlates each person automatically with the scene changes they generate---e.g., shadows, reflections, and motion of loose clothing. The layers can be individually retimed and recombined into a new video, allowing us to achieve realistic, high-quality renderings of retiming effects for real-world videos depicting complex actions and involving multiple individuals, including dancing, trampoline jumping, or group running.

SelfPose: 3D Egocentric Pose Estimation From a Headset Mounted Camera.

We present a new solution to egocentric 3D body pose estimation from monocular images captured from a downward looking fish-eye camera installed on the rim of a head mounted virtual reality device. This unusual viewpoint leads to images with unique visual appearance, characterized by severe self-occlusions and strong perspective distortions that result in a drastic difference in resolution between lower and upper body. We propose a new encoder-decoder architecture with a novel multi-branch decoder designed specifically to account for the varying uncertainty in 2D joint locations. Our quantitative evaluation, both on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric pose estimation approaches. To tackle the severe lack of labelled training data for egocentric 3D pose estimation we also introduced a large-scale photo-realistic synthetic dataset. xR-EgoPose offers 383K frames of high quality renderings of people with diverse skin tones, body shapes and clothing, in a variety of backgrounds and lighting conditions, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of the art results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.

Generalized correlation-based dynamical network analysis: a new high-performance approach for identifying allosteric communications in molecular dynamics trajectories.

Molecular interactions are essential for regulation of cellular processes from the formation of multi-protein complexes to the allosteric activation of enzymes. Identifying the essential residues and molecular features that regulate such interactions is paramount for understanding the biochemical process in question, allowing for suppression of a reaction through drug interventions or optimization of a chemical process using bioengineered molecules. In order to identify important residues and information pathways within molecular complexes, the dynamical network analysis method was developed and has since been broadly applied in the literature. However, in the dawn of exascale computing, this method is frequently limited to relatively small biomolecular systems. In this work, we provide an evolution of the method, application, and interface. All data processing and analysis are conducted through Jupyter notebooks, providing automatic detection of important solvent and ion residues, an optimized and parallel generalized correlation implementation that is linear with respect to the number of nodes in the system, and subsequent community clustering, calculation of betweenness of contacts, and determination of optimal paths. Using the popular visualization program visual molecular dynamics (VMD), high-quality renderings of the networks over the biomolecular structures can be produced. Our new implementation was employed to investigate three different systems, with up to 2.5M atoms, namely, the OMP-decarboxylase, the leucyl-tRNA synthetase complexed with its cognate tRNA and adenylate, and respiratory complex I in a membrane environment. Our enhanced and updated protocol provides the community with an intuitive and interactive interface, which can be easily applied to large macromolecular complexes.

Downsampling scattering parameters for rendering anisotropic media

Volumetric micro-appearance models have provided remarkably high-quality renderings, but are highly data intensive and usually require tens of gigabytes in storage. When an object is viewed from a distance, the highest level of detail offered by these models is usually unnecessary, but traditional linear downsampling weakens the object's intrinsic shadowing structures and can yield poor accuracy. We introduce a joint optimization of single-scattering albedos and phase functions to accurately downsample heterogeneous and anisotropic media. Our method is built upon scaled phase functions , a new representation combining abledos and (standard) phase functions. We also show that modularity can be exploited to greatly reduce the amortized optimization overhead by allowing multiple synthesized models to share one set of down-sampled parameters. Our optimized parameters generalize well to novel lighting and viewing configurations, and the resulting data sets offer several orders of magnitude storage savings.

MoFlow: visualizing conformational changes in molecules as molecular flow improves understanding.

BackgroundCurrent visualizations of molecular motion use a Timeline-analogous representation that conveys "first the molecule was shaped like this, then like this...". This scheme is orthogonal to the Pathline-like human understanding of motion "this part of the molecule moved from here to here along this path". We present MoFlow, a system for visualizing molecular motion using a Pathline-analogous representation.ResultsThe MoFlow system produces high-quality renderings of molecular motion as atom pathlines, as well as interactive WebGL visualizations, and 3D printable models. In a preliminary user study, MoFlow representations are shown to be superior to canonical representations for conveying molecular motion.ConclusionsPathline-based representations of molecular motion are more easily understood than timeline representations. Pathline representations provide other advantages because they represent motion directly, rather than representing structure with inferred motion.

Live small-animal X-ray lung velocimetry and lung micro-tomography at the Australian Synchrotron Imaging and Medical Beamline.

The high flux and coherence produced at long synchrotron beamlines makes them well suited to performing phase-contrast X-ray imaging of the airways and lungs of live small animals. Here, findings of the first live-animal imaging on the Imaging and Medical Beamline (IMBL) at the Australian Synchrotron are reported, demonstrating the feasibility of performing dynamic lung motion measurement and high-resolution micro-tomography. Live anaesthetized mice were imaged using 30 keV monochromatic X-rays at a range of sample-to-detector propagation distances. A frame rate of 100 frames s(-1) allowed lung motion to be determined using X-ray velocimetry. A separate group of humanely killed mice and rats were imaged by computed tomography at high resolution. Images were reconstructed and rendered to demonstrate the capacity for detailed, user-directed display of relevant respiratory anatomy. The ability to perform X-ray velocimetry on live mice at the IMBL was successfully demonstrated. High-quality renderings of the head and lungs visualized both large structures and fine details of the nasal and respiratory anatomy. The effect of sample-to-detector propagation distance on contrast and resolution was also investigated, demonstrating that soft tissue contrast increases, and resolution decreases, with increasing propagation distance. This new capability to perform live-animal imaging and high-resolution micro-tomography at the IMBL enhances the capability for investigation of respiratory diseases and the acceleration of treatment development in Australia.

Virus Maturation and the VIPERdb Virus Structural Database

VIPERdb (Carrillo-Tripp, M., Shepherd, C. M., Borelli, I. A., Venkataraman, S., Lander, G., Natarajan, P., Johnson, J. E., Brooks, C. L., 3rd, and Reddy, V. S. 2009. VIPERdb2: an enhanced and web API enabled relational database for structural virology. Nucleic Acids Res 37:D436-42) is a database for icosahedral virus capsid structures. The emphasis of the resource is on providing data from structural and computational analyses on these systems, as well as high quality renderings for visual exploration. The web site includes powerful search utilities and useful database interface tools. Here we use VIPERdb to address the dynamic character of a virus by relating cryoEM based models of maturation intermediates and associated variance analysis to the high resolution coordinates in VIPERdb for the T=4 Nudaurelia Capensis ω Virus. The outcome is a structure-based description of the maturation energy landscape at near atomic resolution (Tang, J., Kearney, B., Wang, Q., Doerschuk, P. C., Baker, T. S., and Johnson, J. E. 2013. Geometric and Dynamic Analyses of Nudaurelia capensis ω Virus Maturation Reveal the Energy Landscape of Particle Transitions. J. Mol. Recog. Accepted for publication).

Advanced Interactive Preintegrated Volume Rendering with a Power Series

Preintegrated volume rendering produces high-quality renderings without increased sampling rates. However, a look-up table of a conventional preintegrated volume rendering requires a dimensionality of two, which disturbs interactive renderings when the transfer function is changed. Furthermore, as the resolution of the volume data set increases, the memory space required is impractical or inefficient, especially on GPUs. In the past, several approximation methods have been proposed to reduce the complexity of both the time and memory requirement, but most of them do not correctly present thin opaque structures within slabs and ignore the self-attenuation. We propose an advanced interactive preintegrated volume rendering algorithm that achieves not only high-quality renderings comparable to the conventional ones, but also $(O(n))$ time and memory space requirements even with the self-attenuation within the slabs applied. The algorithm proposed in this paper decomposes the exponential term of the ray integration equation into a power series of a finite order in the form of a linear combination to build a one-dimensional look-up table. Moreover, the proposed algorithm effectively applies the self-attenuation that is caused by fully opaque isosurfaces, by introducing an opaque prediction table. Experimental results demonstrate that the proposed algorithm offers renderings visibly identical to existing preintegrated volume renderings without degrading rendering speed.

Object-Based Rendering and 3-D Reconstruction Using a Moveable Image-Based System

This paper proposes a movable image-based rendering (M-IBR) system for improving the viewing freedom and environmental modeling capability of conventional static IBR systems. The system supports object-based rendering and 3-D reconstruction capability and consists of three main components. 1) An improved video stabilization method to reduce the shaky motion frequently encountered in movable IBR systems. It employs local polynomial regression (LPR) to automatically select an appropriate bandwidth for smoothing the estimated motion. 2) A novel view synthesis algorithm using a new segmentation and mutual-information (MI)-based algorithm for dense depth map estimation, which relies on segmentation, LPRbased depth map smoothing, and MI-based matching algorithm to iteratively estimate the depth map. The method is very flexible and both semiautomatic and automatic segmentation methods can be employed. They rank fourth and sixth, respectively, in the Middlebury comparison of existing depth estimation methods. This allows high-quality renderings of outdoor scenes with improved mobility/freedom to be obtained. 3) A new 3-D reconstruction algorithm that utilizes the sequential structure-from-motion technique and the dense depth maps estimated previously. It relies on a new iterative point cloud refinement algorithm based on Kalman filter for outlier removal and the segmentation-MI-based algorithm to further refine the correspondences and the projection matrices. The mobility of our system allows us to recover more conveniently 3-D model of static objects from the improved point cloud using a new robust radial basis function-based modeling algorithm to further suppress possible outliers and generate smooth 3-D meshes of objects. Experimental results show that the proposed 3-D reconstruction algorithm significantly reduces the adverse effect of the outliers and produces high-quality renderings using shadow light field and the model reconstructed.

Diffusion curve textures for resolution independent texture mapping

We introduce a vector representation called diffusion curve textures for mapping diffusion curve images (DCI) onto arbitrary surfaces. In contrast to the original implicit representation of DCIs [Orzan et al. 2008], where determining a single texture value requires iterative computation of the entire DCI via the Poisson equation, diffusion curve textures provide an explicit representation from which the texture value at any point can be solved directly, while preserving the compactness and resolution independence of diffusion curves. This is achieved through a formulation of the DCI diffusion process in terms of Green's functions. This formulation furthermore allows the texture value of any rectangular region (e.g. pixel area) to be solved in closed form, which facilitates anti-aliasing. We develop a GPU algorithm that renders anti-aliased diffusion curve textures in real time, and demonstrate the effectiveness of this method through high quality renderings with detailed control curves and color variations.

Data Driven Surface Reflectance from Sparse and Irregular Samples

AbstractIn recent years, measuring surface reflectance has become an established method for high quality renderings. In this context, especially non‐parametric representations got a lot of attention as they allow for a very accurate representation of complex reflectance behavior. However, the acquisition of this data is a challenging task especially if complex object geometry is involved. Capturing images of the object under varying illumination and view conditions results in irregular angular samplings of the reflectance function with a limited angular resolution. Classical data‐driven techniques, like tensor factorization, are not well suited for such data sets as they require a resampling of the high dimensional measurement data to a regular grid. This grid has to be on a much higher angular resolution to avoid resampling artifacts which in turn would lead to data sets of enormous size. To overcome these problems we introduce a novel, compact data‐driven representation of reflectance functions based on a sum of separable functions which are fitted directly to the irregular set of data without any further resampling. The representation allows for efficient rendering and is also well suited for GPU applications. By exploiting spatial coherence of the reflectance function over the object a very precise reconstruction even of specular materials becomes possible already with a sparse input sampling. This would be impossible using standard data interpolation techniques. Since our algorithm exclusively operates on the compressed representation, it is both efficient in terms of memory use and computational complexity, depending only sub‐linearly on the size of the fully tabulated data. The quality of the reflectance function is evaluated on synthetic data sets as ground truth as well as on real world measurements.

Process for the 3D virtual reconstruction of a microcultural heritage artifact obtained by synchrotron radiation CT technology using open source and free software

Abstract Computer tomography (CT) technology has greatly contributed to the feasibility and convenience of detecting and visualizing the internal material constitution and geometrical fabrication of museum artifacts. This paper presents a case study of 3D virtual reconstruction for the CT-acquisition-based study of a cultural heritage artifact. It documents the complete procedure, including the preprocessing, segmentation and visualization of the data by providing coarse interactive exploration and integrated high-quality renderings. A parallel aim achieved was to use open source tools and free software for segmentation and visualization, thus providing full transparency of the adopted methodology and 3D visualization methods, and a cost effective solution for ordinary CPU-based PC users. Furthermore, the challenges of the large data volumes involved have been addressed using preprocessing, a segmentation scheme and linked front-to-back management to keep interaction and high-quality rendering available, thus achieving corresponding demands.

Video tapestries with continuous temporal zoom

We present a novel approach for summarizing video in the form of a multiscale image that is continuous in both the spatial domain and across the scale dimension: There are no hard borders between discrete moments in time, and a user can zoom smoothly into the image to reveal additional temporal details. We call these artifacts tapestries because their continuous nature is akin to medieval tapestries and other narrative depictions predating the advent of motion pictures. We propose a set of criteria for such a summarization, and a series of optimizations motivated by these criteria. These can be performed as an entirely offline computation to produce high quality renderings, or by adjusting some optimization parameters the later stages can be solved in real time, enabling an interactive interface for video navigation. Our video tapestries combine the best aspects of two common visualizations, providing the visual clarity of DVD chapter menus with the information density and multiple scales of a video editing timeline representation. In addition, they provide continuous transitions between zoom levels. In a user study, participants preferred both the aesthetics and efficiency of tapestries over other interfaces for visual browsing.

Video tapestries with continuous temporal zoom

We present a novel approach for summarizing video in the form of a multiscale image that is continuous in both the spatial domain and across the scale dimension: There are no hard borders between discrete moments in time, and a user can zoom smoothly into the image to reveal additional temporal details. We call these artifacts tapestries because their continuous nature is akin to medieval tapestries and other narrative depictions predating the advent of motion pictures. We propose a set of criteria for such a summarization, and a series of optimizations motivated by these criteria. These can be performed as an entirely offline computation to produce high quality renderings, or by adjusting some optimization parameters the later stages can be solved in real time, enabling an interactive interface for video navigation. Our video tapestries combine the best aspects of two common visualizations, providing the visual clarity of DVD chapter menus with the information density and multiple scales of a video editing timeline representation. In addition, they provide continuous transitions between zoom levels. In a user study, participants preferred both the aesthetics and efficiency of tapestries over other interfaces for visual browsing.

Rendering surface details with diffusion curves

Diffusion curve images (DCI) provide a powerful tool for efficient 2D image generation, storage and manipulation. A DCI consist of curves with colors defined on either side. By diffusing these colors over the image, the final result includes sharp boundaries along the curves with smoothly shaded regions between them. This paper extends the application of diffusion curves to render high quality surface details on 3D objects. The first extension is a view dependent warping technique that dynamically reallocates texture space so that object parts that appear large on screen get more texture for increased detail. The second extension is a dynamic feature embedding technique that retains crisp, anti-aliased curve details even in extreme closeups. The third extension is the application of dynamic feature embedding to displacement mapping and geometry images. Our results show high quality renderings of diffusion curve textures, displacements, and geometry images, all rendered interactively.