Photorealistic Reconstruction Research Articles

The Perception of Depth and Form in Fine Arts, Video Games and Medicine.

Medical imaging technology has greatly improved over the last 40 years. A good example of this is the improved 3D reconstruction capabilities of CTs, which enable photorealistic reconstructions. This technological advancement has also taken place in the computer industry, and the development of modern graphics cards has fueled progress in video games. In art, many of the techniques used to improve three-dimensionality have been known for centuries. At first sight, these fields seem unrelated, but there is actually considerable confluence. This paper focuses on three areas: fine arts, video games and radiology. The art section illustrates different drawing techniques used to create three-dimensionality. The section on video games shows the development of video games from the 1980s to the present day. Finally, the medical imaging section focuses on the first 3D radiography reconstructions in the 1980s and the improvements to the present day. Contemporary video games and 3D reconstructions of CTs enable photorealistic impressions of 3D anatomical structures. An important factor is shadow casting and light, which has been known in fine arts for centuries. The effect on the observer of photorealistic 3D reconstructions in video games and in medical imaging can be largely explained by the knowledge of 3D techniques in drawings and paintings.

3D Snapshot: Invertible Embedding of 3D Neural Representations in a Single Image.

3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method to optimize three neural planes for representing the original neural rendering model. However, this representation is noisy and visually meaningless. We thus propose a dynamic invertible neural network to embed this noisy representation into a plausible image representation of the scene. We demonstrate promising reconstruction quality quantitatively and qualitatively, by comparing to the original neural rendering model, as well as video-based invertible methods. On the other hand, our method can store dozens of NeRFs with a compact restoration network (5MB), and embedding each 3D scene takes up only 160KB of storage. More importantly, our approach is the first solution that allows embedding a neural rendering model into image representations, which enables applications like creating an interactive 3D model from a printed image in the metaverse.

VR2Gather: A Collaborative, Social Virtual Reality System for Adaptive, Multiparty Real-Time Communication

Virtual Reality telecommunication systems promise to overcome the limitations of current real-time teleconferencing solutions, by enabling a better sense of immersion and fostering more natural interpersonal interactions. Many solutions that currently enable immersive teleconferencing employ synthetic avatars to represent their users. However, photorealistic reconstructions have been shown to increase the sense of presence with respect to synthetic avatars in tele-immersive scenarios. In this paper, we present VR2Gather, a costumizable end-to-end system to transmit volumetric contents in multi-party real-time communication. We present the architecture and evaluate the costs and benefits of using different modules and transport mechanisms in terms of CPU usage, latency, and bandwidth. Moreover, we report the user experience based on applications the system has been used for, and how it was customised to meet the requirements using different acquisition and rendering modules.

Narrating Ancient Roman Heritage through Drawings and Digital Architectural Representation: From Historical Archives, UAV and LIDAR to Virtual-Visual Storytelling and HBIM Projects

One of the main objectives of today’s archaeological sites and museums is the development of research, understood as the interpretation and contextualisation of tangible and intangible cultural heritage to broaden the knowledge and accessibility of archaeological parks often unknown to visitors and the public on a large scale. In this perspective, the Appia Antica Archaeological Park aims to support research in digitising infrastructures and archaeological contexts of high historical and cultural value to plan short- and medium-term preservation and maintenance projects. In this context, unmanned aerial vehicles (UAVs) are tools with enormous potential in survey, inspection and digitisation, providing the basis for the subsequent phases of data interpretation, representation and material analysis. Thanks to the photorealistic reconstruction of dense structure from motion (DSfM) in the application of structural inspections, today it is possible to intercept the geometry and material conditions of small, medium and large structures, reducing the costs of inspections, limiting the interruption of the public and providing professionals and visitors with a better volumetric understanding of the system. However, inserting information that gradually accumulates throughout the process requires advanced 3D digital representation techniques, such as HBIM (historic building information modelling), scan-to-BIM approach and interactive forms, such as virtual and augmented reality (VR-AR). For these reasons, this study summarises the experience and lessons learned from the UAV inspection of three research case studies at archaeological, architectural, and infrastructure scales to increase awareness of the Roman-built heritage.

Diffraction-engineered holography: Beyond the depth representation limit of holographic displays

Holography is one of the most prominent approaches to realize true-to-life reconstructions of objects. However, owing to the limited resolution of spatial light modulators compared to static holograms, reconstructed objects exhibit various coherent properties, such as content-dependent defocus blur and interference-induced noise. The coherent properties severely distort depth perception, the core of holographic displays to realize 3D scenes beyond 2D displays. Here, we propose a hologram that imitates defocus blur of incoherent light by engineering diffracted pattern of coherent light with adopting multi-plane holography, thereby offering real world-like defocus blur and photorealistic reconstruction. The proposed hologram is synthesized by optimizing a wave field to reconstruct numerous varifocal images after propagating the corresponding focal distances where the varifocal images are rendered using a physically-based renderer. Moreover, to reduce the computational costs associated with rendering and optimizing, we also demonstrate a network-based synthetic method that requires only an RGB-D image.

Learned lensless 3D camera.

Single-shot three-dimensional (3D) imaging with compact device footprint, high imaging quality, and fast processing speed is challenging in computational imaging. Mask-based lensless imagers, which replace the bulky optics with customized thin optical masks, are portable and lightweight, and can recover 3D object from a snap-shot image. Existing lensless imaging typically requires extensive calibration of its point spread function and heavy computational resources to reconstruct the object. Here we overcome these challenges and demonstrate a compact and learnable lensless 3D camera for real-time photorealistic imaging. We custom designed and fabricated the optical phase mask with an optimized spatial frequency support and axial resolving ability. We developed a simple and robust physics-aware deep learning model with adversarial learning module for real-time depth-resolved photorealistic reconstructions. Our lensless imager does not require calibrating the point spread function and has the capability to resolve depth and "see-through" opaque obstacles to image features being blocked, enabling broad applications in computational imaging.

Neural actor

We propose Neural Actor (NA), a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method is developed upon recent neural scene representation and rendering works which learn representations of geometry and appearance from only 2D images. While existing works demonstrated compelling rendering of static scenes and playback of dynamic scenes, photo-realistic reconstruction and rendering of humans with neural implicit methods, in particular under user-controlled novel poses, is still difficult. To address this problem, we utilize a coarse body model as a proxy to unwarp the surrounding 3D space into a canonical pose. A neural radiance field learns pose-dependent geometric deformations and pose- and view-dependent appearance effects in the canonical space from multi-view video input. To synthesize novel views of high-fidelity dynamic geometry and appearance, NA leverages 2D texture maps defined on the body model as latent variables for predicting residual deformations and the dynamic appearance. Experiments demonstrate that our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses. Furthermore, our method also supports shape control on the free-view synthesis of human actors.

Photorealistic Reconstruction of Visual Texture From EEG Signals.

Recent advances in brain decoding have made it possible to classify image categories based on neural activity. Increasing numbers of studies have further attempted to reconstruct the image itself. However, because images of objects and scenes inherently involve spatial layout information, the reconstruction usually requires retinotopically organized neural data with high spatial resolution, such as fMRI signals. In contrast, spatial layout does not matter in the perception of “texture,” which is known to be represented as spatially global image statistics in the visual cortex. This property of “texture” enables us to reconstruct the perceived image from EEG signals, which have a low spatial resolution. Here, we propose an MVAE-based approach for reconstructing texture images from visual evoked potentials measured from observers viewing natural textures such as the textures of various surfaces and object ensembles. This approach allowed us to reconstruct images that perceptually resemble the original textures with a photographic appearance. The present approach can be used as a method for decoding the highly detailed “impression” of sensory stimuli from brain activity.

Dynamic Multi-Sensor Platform for Efficient Three-Dimensional-Digitalization of Cities

Abstract Three-dimensional urban reconstruction requires the combination of data from different sensors, such as cameras, inertial systems, GPS, and laser sensors. In this technical report, a complete system for the generation of textured volumetric global maps (deep vision) is presented. Our acquisition platform is terrestrial and moves through different urban environments digitizing them. The report is focused on describing the three main problems identified in this type of works. (1) The acquisition of three-dimensional data with high precision, (2) the extraction of the texture and its correlation with the 3D data, and (3) the generation of the surfaces that describe the components of the urban environment. It also describes the methods implemented to extrinsically calibrate the acquisition platform, as well as the methods developed to eliminate the radial and tangential image distortion; and the subsequent generation of a panoramic image. Procedures are developed for the sampling of 3D data and its smoothing. Subsequently, the process to generate textured global maps with a negligible uncertainty is developed and the results are presented. Finally, the process of surface generation and the post-process of eliminating certain holes/occlusions in the meshes are reported. In each section, results obtained are shown. Using the methods presented here for geometric and photorealistic reconstruction of urban environments, high-quality 3D models are generated. The results achieved the following objectives: generate global textured models that preserve the geometry of the scanned scenes.

FlatNet: Towards Photorealistic Scene Reconstruction From Lensless Measurements.

Lensless imaging has emerged as a potential solution towards realizing ultra-miniature cameras by eschewing the bulky lens in a traditional camera. Without a focusing lens, the lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, the current iterative-optimization-based reconstruction algorithms produce noisier and perceptually poorer images. In this work, we propose a non-iterative deep learning-based reconstruction approach that results in orders of magnitude improvement in image quality for lensless reconstructions. Our approach, called FlatNet, lays down a framework for reconstructing high-quality photorealistic images from mask-based lensless cameras, where the camera's forward model formulation is known. FlatNet consists of two stages: (1) an inversion stage that maps the measurement into a space of intermediate reconstruction by learning parameters within the forward model formulation, and (2) a perceptual enhancement stage that improves the perceptual quality of this intermediate reconstruction. These stages are trained together in an end-to-end manner. We show high-quality reconstructions by performing extensive experiments on real and challenging scenes using two different types of lensless prototypes: one which uses a separable forward model and another, which uses a more general non-separable cropped-convolution model. Our end-to-end approach is fast, produces photorealistic reconstructions, and is easy to adopt for other mask-based lensless cameras.

SiGAN: Siamese Generative Adversarial Network for Identity-Preserving Face Hallucination.

Though generative adversarial networks (GANs) can hallucinate high-quality high-resolution (HR) faces from low-resolution (LR) faces, they cannot ensure identity preservation during face hallucination, making the HR faces difficult to recognize. To address this problem, we propose a Siamese GAN (SiGAN) to reconstruct HR faces that visually resemble their corresponding identities. On top of a Siamese network, the proposed SiGAN consists of a pair of two identical generators and one discriminator. We incorporate reconstruction error and identity label information in the loss function of SiGAN in a pairwise manner. By iteratively optimizing the loss functions of the generator pair and the discriminator of SiGAN, we not only achieve visually-pleasing face reconstruction but also ensure that the reconstructed information is useful for identity recognition. Experimental results demonstrate that SiGAN significantly outperforms existing face hallucination GANs in objective face verification performance while achieving promising visual-quality reconstruction. Moreover, for input LR faces with unseen identities that are not part of the training dataset, SiGAN can still achieve reasonable performance.

Geomatics and augmented reality experiments for the cultural heritage

For years, the Laboratory of Geomatics of the Mediterranean University of Reggio Calabria has undertaken an interdisciplinary project for the recovery and dissemination of information regarding the cultural-artistic and archeological heritage of the metropolitan area. The combined use of geomatics technologies (laser scanners, GPS positioning, digital photogrammetry, remote sensing, GPR) allows on the one hand to investigate objects and artifacts, providing metric, form, and location information; and on the other, to catalog information and make it accessible to the community. Indeed, the digitalization and reconstruction tools of 3D models can be the answer to the limits related to communicability in the archeological sector. Precision, detail, and very accurate photo-realistic reconstructions are particularly useful for virtual and augmented reality applications, integrating them in the devices used on a daily basis. The present note concerns, therefore, the acquisition of information using the point cloud from UAVs and laser scanners, the subsequent 3D modeling, and their representation in an augmented reality (AR) environment using mobile platforms. The application was tested on the church of Sant’Antonio Abate, located in the North of Reggio Calabria, which according to studies is the only evidence of medieval architecture in the territory of Reggio Calabria.

CT evaluation of musculoskeletal trauma: initial experience with cinematic rendering.

Multi-detector computed tomography (MDCT) scanners that can quickly acquire volumetric datasets composed of isotropic voxels laid the groundwork for the widespread clinical implementation of 3D MDCT reconstructions, with maximum intensity projection (MIP) and volumetric rendering (VR) becoming important parts of the imaging evaluation of patients with a wide variety of pathologic conditions. Recently, a new 3D reconstruction technique known as cinematic rendering (CR) has become available and is now U.S. FDA approved. CR bears fundamental similarities to VR, but utilizes a more complex lighting model to bring about photorealistic reconstructions. While a tremendous amount of work remains to be done in order to understand the advantages and disadvantages of CR in comparison to traditional 3D reconstruction methods, the images themselves are strikingly detailed and can be interactively manipulated to highlight a variety of different tissue types and anatomic structures. In the following pictorial essay, we provide a number of clinical examples of the use of CR in musculoskeletal imaging, including the evaluation of complex fractures, the delineation of the relationship of fractures to adjacent vasculature and overlying soft tissues, and the visualization of vascular and soft tissue injuries.

3D VISUALIZATION AND PHOTO-REALISTIC RECONSTRUCTION OF THE GREAT TEMPLE OF BEL

Abstract. The Great Temple of Bel in Palmyra was a unique edifice which had blended the well established lines of Greco-Roman architecture with the art and taste of the Orient. With the gilded bronze capitals of its 41 Corinthian columns it was the product of enormous effort and budget. It was the gem of a remarkable epoch of wealthy Palmyra and mighty Roma. With its splendidly decorated adyta ceilings it became a source of inspiration and imagination for Western architecture and decorative arts. While continuing to captivate the World, it was leveled and vanished as a grim result of conflict based vandalism. The aim of this work is to piece together this, the most eloquent and stupendous monument of the Roman East, from its ruins and reconstruct it as it was once extant. Its loss is irreplacable, but its photo-realistic reconstruction can offer some solace by waking the memories of the great temple as in the past. The lost reality of the Great Temple of Bel is revived here by digitally constructing its “ghost images".

REGION-BASED 3D SURFACE RECONSTRUCTION USING IMAGES ACQUIRED BY LOW-COST UNMANNED AERIAL SYSTEMS

Abstract. Accurate 3D surface reconstruction of our environment has become essential for an unlimited number of emerging applications. In the past few years, Unmanned Aerial Systems (UAS) are evolving as low-cost and flexible platforms for geospatial data collection that could meet the needs of aforementioned application and overcome limitations of traditional airborne and terrestrial mobile mapping systems. Due to their payload restrictions, these systems usually include consumer-grade imaging and positioning sensor which will negatively impact the quality of the collected geospatial data and reconstructed surfaces. Therefore, new surface reconstruction surfaces are needed to mitigate the impact of using low-cost sensors on the final products. To date, different approaches have been proposed to for 3D surface construction using overlapping images collected by imaging sensor mounted on moving platforms. In these approaches, 3D surfaces are mainly reconstructed based on dense matching techniques. However, generated 3D point clouds might not accurately represent the scanned surfaces due to point density variations and edge preservation problems. In order to resolve these problems, a new region-based 3D surface renostruction trchnique is introduced in this paper. This approach aims to generate a 3D photo-realistic model of individually scanned surfaces within the captured images. This approach is initiated by a Semi-Global dense Matching procedure is carried out to generate a 3D point cloud from the scanned area within the collected images. The generated point cloud is then segmented to extract individual planar surfaces. Finally, a novel region-based texturing technique is implemented for photorealistic reconstruction of the extracted planar surfaces. Experimental results using images collected by a camera mounted on a low-cost UAS demonstrate the feasibility of the proposed approach for photorealistic 3D surface reconstruction.

Simple wave-field rendering for photorealistic reconstruction in polygon-based high-definition computer holography

A simple and practical technique is presented for creating fine three-dimensional (3D) images with polygon-based computer-generated holograms. The polygon-based method is a technique for computing the optical wave-field of virtual 3D scenes given by a numerical model. The presented method takes less computation time than common point-source methods and produces fine spatial 3D images of deep 3D scenes that convey a strong sensation of depth, unlike conventional 3D systems providing only binocular disparity. However, smooth surfaces cannot be reconstructed using the presented method because the surfaces are approximated by planar polygons. This problem is resolved by introducing a simple rendering technique that is almost the same as that in common computer graphics, since the polygon-based method has similarity to rendering techniques in computer graphics. Two actual computer holograms are presented to verify and demonstrate the proposed technique. One is a hologram of a live face whose shape is measured using a 3D laser scanner that outputs polygon-mesh data. The other is for a scene including the moon. Both are created employing the proposed rendering techniques of the texture mapping of real photographs and smooth shading.

Hologram synthesis for photorealistic reconstruction

Computation of diffraction patterns, and thus holograms, of scenes with photorealistic properties is a highly complicated and demanding process. An algorithm, based primarily on computer graphics methods, for computing full-parallax diffraction patterns of complicated surfaces with realistic texture and reflectivity properties is proposed and tested. The algorithm is implemented on single-CPU, multiple-CPU and GPU platforms. An alternative algorithm, which implements reduced occlusion diffraction patterns for much faster but somewhat lower quality results, is also developed and tested. The algorithms allow GPU-aided calculations and easy parallelization. Both numerical and optical reconstructions are conducted. The results indicate that the presented algorithms compute diffraction patterns that provide successful photorealistic reconstructions; the computation times are acceptable especially on the GPU implementations.

Real-time exploration and photorealistic reconstruction of large natural environments

This paper presents a hybrid (geometry- and image-based) framework suitable for providing photorealistic walkthroughs of large, complex outdoor scenes, based only on a small set of real images from the scene. To this end, a novel data representation of a 3D scene is proposed, which is called morphable 3D panoramas. Motion is assumed to be taking place along a predefined path of the 3D environment and the input to the system is a sparse set of stereoscopic views at certain positions (key positions) along that path (one view per position). An approximate local 3D model is constructed from each view, capable of capturing the photometric and geometric properties of the scene only locally. Then, during the rendering process, a continuous morphing (both photometric as well as geometric) takes place between successive local 3D models, using what we call a ‘morphable 3D model’. For the estimation of the photometric morphing, a robust algorithm capable of extracting a dense field of 2D correspondences between wide-baseline images is used, whereas, for the geometric morphing, a novel method of computing 3D correspondences between local models is proposed. In this way, a physically valid morphing is always produced, which is thus kept transparent from the user. Moreover, a highly optimized rendering path is used during morphing. Thanks to the use of appropriate pixel and vertex shaders, this rendering path can be run fully in 3D graphics hardware and thus allows for high frame rates. Our system can be extended to handle multiple stereoscopic views (and therefore multiple local models) per key position of the path (related by a camera rotation). In this case, one local 3D panorama (per key position) is constructed, comprising all local 3D models therein, and so a ‘morphable 3D panorama’ is now used during the rendering process. For handling the geometric consistency of each 3D panorama, a technique which is based on solving a partial differential equation is adopted. The effectiveness of our framework is demonstrated by using it for the 3D visual reconstruction of the Samaria Gorge in Crete.

Automatic 3D model reconstruction based on novel pose estimation and integration techniques

An automatic three-dimensional (3D) model reconstruction technique is presented to acquire complete and closed 3D models of real objects. The technique is based on novel approaches to pose estimation and integration. Two different poses of an object are used because a single pose often hides some surfaces from a range sensor. A second pose is used to expose such surfaces to the sensor. Two partial 3D models are reconstructed for two different poses of the object using a multi-view 3D modeling technique. The two 3D models are then registered in two steps—coarse registration, and its refinement. Coarse registration is facilitated by a novel pose estimation technique, which estimates a rigid transformation between two models. The pose is estimated by matching a stable tangent plane (STP) of each pose-model with the base tangent plane, which is invariant for a vision system. We employ geometric constraints to find the STP. After registration refinement, two models are integrated to a complete 3D model based on voxel classification defined in multi-view integration. Texture mapping is done to obtain a photo-realistic reconstruction of the object. Reconstruction results and error analysis are presented for several real objects.