Photorealistic Textures Research Articles

3D scanning in Cultural Heritage: model creation and digital restoration of the tombstone of the bishop Vasilije Petrović (Negosh) in Saint Petersburg

3D laser and optical scanning techniques are widely employed in Cultural Heritage preservation for documenting pieces of different sizes and shapes. However, due to limitations of the methods available today, the results obtained vary according to different characteristics of the object under study. External surfaces showing very smooth features make the capturing process and the posterior processing a more elaborate task, requiring special attention and additional retouching to improve the final 3D model. Tombstones are a typical example of this type of objects, combining both flat external contours and refined outlines. In this work the procedure of creation of the digital model of the tombstone of the bishop Vasilije Petrović (Negosh) is described, a piece of art which combines both technical and historical importance. Post-processing steps required involved different software (Geomagic Design X, ZBrush, Adobe Photoshop) for surface correction and cleaning, details and photorealistic texture improvements and lost material restoration.

TwinTex: Geometry-Aware Texture Generation for Abstracted 3D Architectural Models

Coarse architectural models are often generated at scales ranging from individual buildings to scenes for downstream applications such as Digital Twin City, Metaverse, LODs, etc. Such piece-wise planar models can be abstracted as twins from 3D dense reconstructions. However, these models typically lack realistic texture relative to the real building or scene, making them unsuitable for vivid display or direct reference. In this paper, we present TwinTex , the first automatic texture mapping framework to generate a photorealistic texture for a piece-wise planar proxy. Our method addresses most challenges occurring in such twin texture generation. Specifically, for each primitive plane, we first select a small set of photos with greedy heuristics considering photometric quality, perspective quality and facade texture completeness. Then, different levels of line features (LoLs) are extracted from the set of selected photos to generate guidance for later steps. With LoLs, we employ optimization algorithms to align texture with geometry from local to global. Finally, we fine-tune a diffusion model with a multi-mask initialization component and a new dataset to inpaint the missing region. Experimental results on many buildings, indoor scenes and man-made objects of varying complexity demonstrate the generalization ability of our algorithm. Our approach surpasses state-of-the-art texture mapping methods in terms of high-fidelity quality and reaches a human-expert production level with much less effort.

Towards High-Fidelity Face Self-Occlusion Recovery via Multi-View Residual-Based GAN Inversion

Face self-occlusions are inevitable due to the 3D nature of the human face and the loss of information in the projection process from 3D to 2D images. While recovering face self-occlusions based on 3D face reconstruction, e.g., 3D Morphable Model (3DMM) and its variants provides an effective solution, most of the existing methods show apparent limitations in expressing high-fidelity, natural, and diverse facial details. To overcome these limitations, we propose in this paper a new generative adversarial network (MvInvert) for natural face self-occlusion recovery without using paired image-texture data. We design a coarse-to-fine generator for photorealistic texture generation. A coarse texture is computed by inpainting the invisible areas in the photorealistic but incomplete texture sampled directly from the 2D image using the unrealistic but complete statistical texture from 3DMM. Then, we design a multi-view Residual-based GAN Inversion, which re-renders and refines multi-view 2D images, which are used for extracting multiple high-fidelity textures. Finally, these high-fidelity textures are fused based on their visibility maps via Poisson blending. To perform adversarial learning to assure the quality of the recovered texture, we design a discriminator consisting of two heads, i.e., one for global and local discrimination between the recovered texture and a small set of real textures in UV space, and the other for discrimination between the input image and the re-rendered 2D face images via pixel-wise, identity, and adversarial losses. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods in face self-occlusion recovery under unconstrained scenarios.

Lost archaeological heritage: virtual reconstruction of the medieval castle of San Salvador de Todea

The medieval castle of San Salvador de Todea constitutes a remarkable paradigm for the rock castles of Galicia and the northwest of Spain, attending to its features but also to the destruction it was subjected. The hill where it stands was the object of various archaeological interventions led from 2016 to 2018, that identified the site as a medieval fortress and recovered part of its structures. Being one of few medieval castles of the 12th-15th centuries in Galicia subjected to consecutive interventions and actively studied. Although the excavations provided an important amount of information about the chronology, distribution, material culture and even diet of the inhabitants, the interpretation of the castle is really complex. The preserved remains correspond to the foundations of the structures, mainly a few walls and a high amount of carvings scattered over the hill. This paper gathers the results of the investigation developed to create a reconstructive hypothesis for the castle and its materialization in a digital model, as a way to interpret and understand its structure. To achieve this aim, we combined the analysis of the archaeological data and the geomorphology of the own hill, with the study of fortresses with similar disposition and chronology, and the appliance of architectonical rules, in a multidisciplinary study. In order to generate a reasonable, justified and scientific model of the castle, useful as an instrument to study and comprehend this type of construction. The resulting model shows the disposition, height and entity of the structures that conformed to the castle, the transit areas and indoor spaces, and provides new data to calculate the visibility and predominance of the construction over the landscape, among others. Finally, to ensure the rigour and scientific approach of the reconstruction, we utilised a scale of certainty level to show the results, along with photorealistic textures to recreate the probable aspect of the castle. Highlights: - Virtual reconstruction of a Late Medieval rock castle, from the archaeological remains recorded during the excavations of the site and the comparison with similar structures. - Overview of the reconstruction methodology, decision-making process and resulting constructive hypothesis, with a detailed description of the level of certainty of each part. - Multidisciplinary approach to achieve a scientific model that helps to understand the castle distribution and its importance on the control of the landscape.

Digital 2D, 2.5D and 3D Methods for Adding Photo-Realistic Textures to 3D Facial Depictions of People from the Past.

Facial reconstruction is a technique that can be used to estimate individual faces from human skulls. The presentation of 3D facial reconstructions as photo-realistic depictions of people from the past to public audiences varies widely due to differing methods, the artists' CGI skillset, and access to VFX software required to generate plausible faces.This chapter describes three digital methods for the addition of realistic textures to 3D facial reconstructions; a 2D photo-composite method, a 3D digital painting and rendering method, and a previously undescribed hybrid 2.5D method.These methods are compared and discussed in relation to artistic proficiency, morphological accuracy and practitioner bias.

Augmented reality in the dynamic world of virtual tourism

The issue of virtual tourism penetrates the consciousness of many current explorers and visitors to interesting sites and local attractions. Is it really necessary to physically travel to the chosen location, sacrifice time, money and increase your carbon footprint? The article deals with the issue of creating an application using augmented reality for the presentation of 3D models of cultural monuments. The application is designed for mobile devices, on which museum visitors can view digital copies of selected objects of interest. They can also take a virtual tour of the museum through their mobile devices from the comfort of their home. The introductory part of the article describes surveying of shape and visual properties in situ using photogrammetry and laser scanning. The sequent chapters are an analysis of the creation of 3D models in real scale with photorealistic textures and their presentation in a mixed reality environment. In our case, we will specifically focus on the augmented reality application and its development in the Unity multiplatform game engine. The article aims to describe the development of a mobile marker-based AR application briefly, to demonstrate the possibilities of augmented reality and to indicate the potential benefits resulting from the integration of X-reality into our lives. Specifically, we will focus on the application of AR to cartographic works and the enrichment of the content of classical 2D map works and plans with virtually designed 3D models on their surface.

Unpaired Remote Sensing Image Super-Resolution with Multi-Stage Aggregation Networks

Mainstream image super-resolution (SR) methods are generally based on paired training samples. As the high-resolution (HR) remote sensing images are difficult to collect with a limited imaging device, most of the existing remote sensing super-resolution methods try to down-sample the collected original images to generate an auxiliary low-resolution (LR) image and form a paired pseudo HR-LR dataset for training. However, the distribution of the generated LR images is generally inconsistent with the real images due to the limitation of remote sensing imaging devices. In this paper, we propose a perceptually unpaired super-resolution method by constructing a multi-stage aggregation network (MSAN). The optimization of the network depends on consistency losses. In particular, the first phase is to preserve the contents of the super-resolved results, by constraining the content consistency between the down-scaled SR results and the low-quality low-resolution inputs. The second stage minimizes perceptual feature loss between the current result and LR input to constrain perceptual-content consistency. The final phase employs the generative adversarial network (GAN) to adding photo-realistic textures by constraining perceptual-distribution consistency. Numerous experiments on synthetic remote sensing datasets and real remote sensing images show that our method obtains more plausible results than other SR methods quantitatively and qualitatively. The PSNR of our network is 0.06dB higher than the SOTA method—HAN on the UC Merced test set with complex degradation.

Synthesis and Visualization of Photorealistic Textures for 3D Face Reconstruction of Prehistoric Human

Reconstruction of face 3D shape and its texture is a challenging task in the modern anthropology. While a skilled anthropologist could reconstruct an appearance of a prehistoric human from its skull, there are no automated methods to date for automatic anthropological face 3D reconstruction and texturing. We propose a deep learning framework for synthesis and visualization of photorealistic textures for 3D face reconstruction of prehistoric human. Our framework leverages a joint face-skull model based on generative adversarial networks. Specifically, we train two image-to-image translation models to separate 3D face reconstruction and texturing. The first model translates an input depth map of a human skull to a possible depth map of its face and its semantic parts labeling. The second model, performs a multimodal translation of the generated semantic labeling to multiple photorealistic textures. We generate a dataset consisting of 3D models of human faces and skulls to train our 3D reconstruction model. The dataset includes paired samples obtained from computed tomography and unpaired samples representing 3D models of skulls of prehistoric human. We train our texture synthesis model on the CelebAMask-HQ dataset. We evaluate our model qualitatively and quantitatively to demonstrate that it provides robust 3D face reconstruction of prehistoric human with multimodal photorealistic texturing.

Uncertainty-Aware Semantic Guidance and Estimation for Image Inpainting

Completing a corrupted image by filling in correct structures and reasonable textures for a complex scene remains an elusive challenge. In case that a missing hole involves diverse semantic information, conventional two-stage approaches based on structural information often lead to unreliable structural prediction and ambiguous visual texture generation. To address the problem, we propose a SEmantic GUidance and Estimation Network (SeGuE-Net) that iteratively evaluates the uncertainty of inpainted visual contents based on pixel-wise semantic inference and optimize structural priors and inpainted contents alternatively. Specifically, SeGuE-Net utilizes semantic segmentation maps as guidance in each iteration of image inpainting, under which location-dependent inferences are re-estimated, and, accordingly, poorly-inferred regions are refined in subsequent iterations. Extensive experiments on real-world images demonstrate the superiority of our proposed method over state-of-the-art approaches in terms of clear boundaries and photo-realistic textures.

The Development of a BMI-Guided Shape Morphing Technique and the Effects of an Individualized Figure Rating Scale on Self-Perception of Body Size

The creation of personalized avatars that may be morphed to simulate realistic changes in body size is useful when studying self-perception of body size. One drawback is that these methods are resource intensive compared to rating scales that rely upon generalized drawings. Little is known about how body perception ratings compare across different methods, particularly across differing levels of personalized detail in visualizations. This knowledge is essential to inform future decisions about the appropriate tradeoff between personalized realism and resource availability. The current study aimed to determine the impact of varying degrees of personalized realism on self-perception of body size. We explored this topic in young adult women, using a generalized line drawing scale, as well as several types of personalized avatars, including 3D textured images presented in immersive virtual reality (VR). Body perception ratings using generalized line drawings were often higher than responses using individualized visualization methods. While the personalized details seemed to help with identification, there were few differences among the three conditions containing different amounts of individualized realism (e.g., photo-realistic texture). These results suggest that using scales based on personalized texture and limb dimensions are beneficial, although presentation in immersive VR may not be essential.

Normalization of face illumination with photorealistic texture via deep image prior synthesis

Illumination normalization for face recognition is of great importance when the face image is taken under a harsh lighting condition and only one single image is available. In this paper, an illumination normalization method that can preserve face identity and make the processed image have photorealistic face texture is proposed. First, an illumination regression filter is proposed to remove a large number of illumination components of the input image. Then, the accelerated proximal gradient algorithm is used for low-rank decomposition of the filtered image, which further reduces the residual illumination and noise of the face image. However, after these two operations, the face image has a peeled-off appearance. To avoid the unrealistic appearance, a deep image prior synthesis process is developed to synthesize the photorealistic face texture. To accomplish this task, the filtered image, the low-rank decomposed image, the original image and any a uniformly illuminated frontal face image as reference are used as synthetic ingredients. These ingredients, as prior knowledge, synthesize a face image that has the texture features of the reference image and retains the identity of the input image. We try to bridge the appearance gap between the synthetic ingredients and the final synthetic image by a deep neural network. Instead of training the network on a large dataset, the final synthetic image is generated by iterating the parameters of the generator-structured network according to a certain proportion of the synthetic ingredients. Systematic evaluations conducted on public databases demonstrate that the proposed method is robust to illumination with better performance than state-of-the-art methods.

Full Face-and-Head 3D Model With Photorealistic Texture

In the recent period, significant progress has been achieved towards reconstructing the 3D face model from face image. With the support of the render engines and sufficient data, the reconstruction results are fine in detail. Nevertheless, the research on the 3D face reconstruction with texture from a single unrestricted face image is imperfect. The rebuild process lacks essential structure and texture information in the profile and the craniofacial region. To address this problem, we present a method of creating a 3D full face-and-head model with photorealistic texture from a single “in-the-wild” face image in this paper. To this end, we introduce a pipeline to integrate the highly-detailed face model into the basic model. Specifically, the basic model was built by multilinear optimization, and the highly-detailed face model which represents the facial features generated by constrained illumination distribution. Additionally, to infer the invisible region texture information corresponding to the input face image, we design an effective architecture with the generative adversarial network (GAN) for panoramic UV texture generation. The final results after UV texture mapping were visualized in the experiment, which demonstrates that the model faithfully recovers the photorealistic details in arbitrary perspective. Furthermore, compared to the state-of-the-art facial modeling techniques and existing commercial solutions, our method takes less input and performs better in surface detail.

COMPARISON OF UAVS PERFORMANCE FOR A ROMAN AMPHITHEATRE SURVEY: THE CASE OF AVELLA (ITALY)

Abstract. In the field of archaeological surveying, remote sensors and especially photogrammetric and laser scanner systems are widely used to create 3D models. The use of photogrammetric surveying with UAVs (Unmanned Aerial Vehicles), combined with Computer Vision algorithms, allows the building of three-dimensional models, characterized by photo-realistic textures. The choice of which method to use mainly depends on the complexity of the investigated site, the accuracy requirements and the available budget and time. The different components of the UAV system determine its characteristics in terms of performance and accuracy, therefore define its quality and the cost too. This study presents an assessment of the accuracy of point clouds derived by two UAV systems, a commercial quadcopter (DJI Phantom 3 Professional), a professional assembled hexacopter, and by a TLS (Terrestrial Laser Scanner) in order to compare photogrammetric and laser scanner data for archaeological applications. In this paper, we present a case study to compare and analyse the metric accuracy of the point clouds and the distribution of the GCPs (Ground Control Points). This accuracy assessment will serve to quantify the uncertainty in the absolute position of the GCPs, identified on the panoramic images in the absence of artificial targets. Executed experiments showed that in tested UAVs, the choice of the GCPs has significant impact on point cloud accuracy. Estimated absolute accuracy of point clouds collected during both test flights was better than 5 cm.

Unsupervised texture reconstruction method using bidirectional similarity function for 3-D measurements

The developments of three-dimensional (3-D) imaging and display technology have created important demands for 3-D model with realistic texture. Due to texture misalignment and inconsistent illumination, texture reconstruction is subject to degradation like blurring, ghosting and color discontinuity. A composite weight texture blending approach is proposed to mitigate the artifacts. For inaccurate geometric model, bidirectional similarity function based multi-scale energy minimization method is proposed to correct texture misalignment. The average processing time of the proposed method is only 40% of existing method. The optical experiments demonstrate that photorealistic texture could be achieved for inaccurate geometric model and inconsistent illumination.

Visual Perception and Evaluation of Photo-Realistic Self-Avatars From 3D Body Scans in Males and Females

The creation or streaming of photo-realistic self-avatars is important for virtual reality applications that aim for perception and action to replicate real world experience. The appearance and recognition of a digital self-avatar may be especially important for applications related to telepresence, embodied virtual reality, or immersive games. We investigated gender differences in the use of visual cues (shape, texture) of a self-avatar for estimating body weight and evaluating avatar appearance. A full-body scanner was used to capture each participant's body geometry and color information and a set of 3D virtual avatars with realistic weight variations was created based on a statistical body model. Additionally, a second set of avatars was created with an average underlying body shape matched to each participant's height and weight. In four sets of psychophysical experiments, the influence of visual cues on the accuracy of body weight estimation and the sensitivity to weight changes was assessed by manipulating body shape (own, average) and texture (own photo-realistic, checkerboard). The avatars were presented on a large-screen display, and participants responded to whether the avatar's weight corresponded to their own weight. Participants also adjusted the avatar's weight to their desired weight and evaluated the avatar's appearance with regard to similarity to their own body, uncanniness, and their willingness to accept it as a digital representation of the self. The results of the psychophysical experiments revealed no gender difference in the accuracy of estimating body weight in avatars. However, males accepted a larger weight range of the avatars as corresponding to their own. In terms of the ideal body weight, females but not males desired a thinner body. With regard to the evaluation of avatar appearance, the questionnaire responses suggest that own photo-realistic texture was more important to males for higher similarity ratings, while own body shape seemed to be more important to females. These results argue for gender-specific considerations when creating self-avatars.

INTEGRATION OF THREE-DIMENSIONAL DIGITAL MODELS AND 3D GIS: THE DOCUMENTATION OF THE MEDIEVAL BURIALS OF AMITERNUM (L’AQUILA, ITALY)

Abstract. This paper illustrates the results of an experimentation carried out by a multi-disciplinary research group made up of researchers from ITC-CNR of L'Aquila and of archaeologists of the University of L'Aquila. The research project carried out by the team is based on the analysis of the archaeological heritage (in particular, the documentation of some burials found in the medieval site of Amiternum, near L'Aquila). This starts from methods based on digital photogrammetric restitution, based on Structure from Motion (SfM) algorithms, and the generation of photorealistic textures in order to manage, in a 3D GIS environment, complex archaeological and anthropological data. The choice of technology to use is often determined by the specific needs of the survey, the purpose of the project, the budget and experience of the researchers, and the geometric characteristics of the assets, rather than the precision to be achieved. For the survey of the archaeological excavation of the medieval site of Amiternum, it was decided to use digital photogrammetry given that the objective was to document, with a rapid survey compatible with the times of an archaeological excavation the phases of image acquisition, processing and post processing of the site model. Furthermore, thanks to the integration of two technologies, digital photogrammetry and GIS, and the undisputed improvement in the management of 3D data by the GIS, three-dimensionality, in archeology in general, has become an indispensable component for site interpretation and for the documentation of the data.

COMPUTER GRAPHICS MEETS IMAGE FUSION: THE POWER OF TEXTURE BAKING TO SIMULTANEOUSLY VISUALISE 3D SURFACE FEATURES AND COLOUR

Abstract. Since a few years, structure-from-motion and multi-view stereo pipelines have become omnipresent in the cultural heritage domain. The fact that such Image-Based Modelling (IBM) approaches are capable of providing a photo-realistic texture along the threedimensional (3D) digital surface geometry is often considered a unique selling point, certainly for those cases that aim for a visually pleasing result. However, this texture can very often also obscure the underlying geometrical details of the surface, making it very hard to assess the morphological features of the digitised artefact or scene. Instead of constantly switching between the textured and untextured version of the 3D surface model, this paper presents a new method to generate a morphology-enhanced colour texture for the 3D polymesh. The presented approach tries to overcome this switching between objects visualisations by fusing the original colour texture data with a specific depiction of the surface normals. Whether applied to the original 3D surface model or a lowresolution derivative, this newly generated texture does not solely convey the colours in a proper way but also enhances the smalland large-scale spatial and morphological features that are hard or impossible to perceive in the original textured model. In addition, the technique is very useful for low-end 3D viewers, since no additional memory and computing capacity are needed to convey relief details properly. Apart from simple visualisation purposes, the textured 3D models are now also better suited for on-surface interpretative mapping and the generation of line drawings.

Rapid Texture Optimization of Three-Dimensional Urban Model Based on Oblique Images.

Seamless texture mapping is one of the key technologies for photorealistic 3D texture reconstruction. In this paper, a method of rapid texture optimization of 3D urban reconstruction based on oblique images is proposed aiming at the existence of texture fragments, seams, and inconsistency of color in urban 3D texture mapping based on low-altitude oblique images. First, we explore implementing radiation correction on the experimental images with a radiation procession algorithm. Then, an efficient occlusion detection algorithm based on OpenGL is proposed according to the mapping relation between the terrain triangular mesh surface and the images to implement the occlusion detection of the visible texture on the triangular facets as well as create a list of visible images. Finally, a texture clustering algorithm is put forward based on Markov Random Field utilizing the inherent attributes of the images and solve the energy function minimization by Graph-Cuts. The experimental results display that the method is capable of decreasing the existence of texture fragments, seams, and inconsistency of color in the 3D texture model reconstruction.

Real-time 3D rendering with hatching

We present an approach for real-time pen-and-ink hatching renderings on large scenes. Starting with 3D models including photorealistic textures and materials, we aim to propose a solution that prod...

LASER SCANNER AND CAMERA-EQUIPPED UAV ARCHITECTURAL SURVEYS

Abstract. The paper reports the workflow of 3D modelling and photorealistic texture mapping based on close range imagery acquired through a terrestrial laser scan and a camera mounted on a mini Unmanned Aerial Vehicle (UAV). The 3D models of the same architectural object – a watchtower named “Torre Pelosa” (Bari, Italy) - were compared in order to evaluate strengths and weaknesses of the used instruments and methods. The evaluated parameters where geo-referencing capability, manageability, manoeuvrability, operational constraint, survey completeness and costs. Despite a lower accuracy of the results, the use of a UAV mounted camera is essential to obtain a whole representation of the tower.