Photorealistic Renderings Research Articles

A Comparative Study of Generative AI Models for Interior Design

Abstract: The integration of Generative AI in interior design has transformed traditional methods,allowing designers to explore new concepts with impressive efficiency. This paper presents a comparative study of leading generative models—StyleGAN, Variational Autoencoders (VAEs), Pix2Pix, and Reinforcement Learning (RL)—evaluating their effectiveness in turning sketches into photorealistic renderings, generating diverse room layouts, and optimizing spaces. By analyzing the results of these models, we show their ability to create unique design solutions that meet functional requirements while enhancing aesthetic appeal. The study highlights substantial enhancements in design precision, emphasizing the potential of generative AI models to elevate the design process and create more tailored interior solutions. This survey examines the methods and performance of each model and looksat future possibilities for using Generative AI to advance the field of interior design.

3D Scene Creation and Rendering via Rough Meshes: A Lighting Transfer Avenue.

This paper studies how to flexibly integrate reconstructed 3D models into practical 3D modeling pipelines such as 3D scene creation and rendering. Due to the technical difficulty, one can only obtain rough 3D models (R3DMs) for most real objects using existing 3D reconstruction techniques. As a result, physically-based rendering (PBR) would render low-quality images or videos for scenes that are constructed by R3DMs. One promising solution would be representing real-world objects as Neural Fields such as NeRFs, which are able to generate photo-realistic renderings of an object under desired viewpoints. However, a drawback is that the synthesized views through Neural Fields Rendering (NFR) cannot reflect the simulated lighting details on R3DMs in PBR pipelines, especially when object interactions in the 3D scene creation cause local shadows. To solve this dilemma, we propose a lighting transfer network (LighTNet) to bridge NFR and PBR, such that they can benefit from each other. LighTNet reasons about a simplified image composition model, remedies the uneven surface issue caused by R3DMs, and is empowered by several perceptual-motivated constraints and a new Lab angle loss which enhances the contrast between lighting strength and colors. Comparisons demonstrate that LighTNet is superior in synthesizing impressive lighting, and is promising in pushing NFR further in practical 3D modeling workflows.

Interactive geometry editing of Neural Radiance Fields

Neural Radiance Fields (NeRF) have recently emerged as a promising approach for synthesizing highly realistic images from 3D scenes. This technology has shown impressive results in capturing intricate details and producing photorealistic renderings. However, one of the limitations of traditional NeRF approaches is the difficulty in editing and manipulating the geometry of the scene once it has been captured. This restriction hinders creative freedom and practical applicability.In this paper, we propose a method that enables interactive geometry editing for neural radiance fields manipulation. We use two proxy cages (inner cage and outer cage) to edit a scene. The inner cage defines the operation target, and the outer cage defines the adjustment space. Various operations apply to the two cages. After cage selection, operations on the inner cage lead to the desired transformation of the inner cage and adjustment of the outer cage. Users can edit the scene with translation, rotation, scaling, or combinations. The operations on the corners and edges of the cage are also supported. Our method does not need any explicit 3D geometry representations. The interactive geometry editing applies directly to the implicit neural radiance fields. Extensive experimental results demonstrate the effectiveness of our approach.

Tire Texture Monitoring (VGG 19 VS Efficient Net b7)

Tires are crucial components of vehicles, continuously in contact with the road. Monitoring tire conditions is vital for safety and performance, as degradation in tire treads and sidewalls can affect traction, fuel efficiency, longevity, and road noise. This research leverages both VGG19 and Efficient Net B7 algorithms to enhance tire image rendering, addressing limitations of traditional techniques. Using a binary classification algorithm, we classify tire images as healthy or cracked. By fine-tuning VGG19 and EfficientNet B7 on a specialized tire dataset, we achieve high-quality, photorealistic renderings. Our results demonstrate remarkable improvements in texture quality and visual realism compared to traditional methods. The rendered images exhibit finer details and more accurate representations of the tire’s tread patterns and material properties. This research contributes to the field of computer graphics by presenting a novel application of deep learning techniques to a specific industrial need, paving the way for future advancements in high-quality rendering of complex tire textures. Key Words: VGG19,Photorealistic rendering, deep learning.

Depth assisted novel view synthesis using few images

In this paper, we introduce a novel approach to improve the performance of Neural Radiance Fields (NeRF) from limited input views. NeRF has exhibited impressive capabilities in producing photo-realistic renderings when trained on dense input views, but its performance degrades as the number of training views decreases. Our key insight is that the original NeRF lacks geometric regularization and appearance information due to limited inputs, resulting in an over-fitting issue. To address this challenge, we present a novel method: first, a global sampling method with geometric regularization is employed by utilizing warped images as additional pseudo-views, which optimizes the multi-view consistency during the training. Second, we introduce a local patch sampling technique with perceptual regularization to ensure pixel correspondence in appearance. Furthermore, we incorporate depth information for explicit geometry regularization. We evaluate our method on the DTU dataset and LLFF dataset from a different number of inputs. Extensive evaluations demonstrate that our approach outperforms existing benchmarks across various metrics, achieving state-of-the-art results.

NeRFail: Neural Radiance Fields-Based Multiview Adversarial Attack

Adversarial attacks, i.e., generating adversarial perturbations with a small magnitude to deceive deep neural networks, are important for investigating and improving model trustworthiness. Traditionally, the topic was scoped within 2D images without considering 3D multiview information. Benefiting from Neural Radiance Fields (NeRF), one can easily reconstruct a 3D scene with a Multi-Layer Perceptron (MLP) from given 2D views and synthesize photo-realistic renderings of novel vantages. This opens up a door to discussing the possibility of undertaking to attack multiview NeRF network with downstream tasks from different rendering angles, which we denote Neural Radiance Fiels-based multiview adversarial Attack (NeRFail). The goal is, given one scene and a subset of views, to deceive the recognition results of agnostic view angles as well as given views. To do so, we propose a transformation mapping from pixels to 3D points such that our attack generates multiview adversarial perturbations by attacking a subset of images with different views, intending to prevent the downstream classifier from correctly predicting images rendered by NeRF from other views. Experiments show that our multiview adversarial perturbations successfully obfuscate the downstream classifier at both known and unknown views. Notably, when retraining another NeRF on the perturbed training data, we show that the perturbation can be inherited and reproduced. The code can be found at https://github.com/jiang-wenxiang/NeRFail.

Head3D: Complete 3D Head Generation via Tri-plane Feature Distillation

Head generation with diverse identities is an important task in computer vision and computer graphics, widely used in multimedia applications. However, current full-head generation methods require a large number of three-dimensional (3D) scans or multi-view images to train the model, resulting in expensive data acquisition costs. To address this issue, we propose Head3D, a method to generate full 3D heads with limited multi-view images. Specifically, our approach first extracts facial priors represented by tri-planes learned in EG3D, a 3D-aware generative model, and then proposes feature distillation to deliver the 3D frontal faces within complete heads without compromising head integrity. To mitigate the domain gap between the face and head models, we present a dual-discriminator to guide the frontal and back head generation. Our model achieves cost-efficient and diverse complete head generation with photo-realistic renderings and high-quality geometry representations. Extensive experiments demonstrate the effectiveness of our proposed Head3D, both qualitatively and quantitatively.

Unsupervised Point Cloud Representation Learning by Clustering and Neural Rendering

Data augmentation has contributed to the rapid advancement of unsupervised learning on 3D point clouds. However, we argue that data augmentation is not ideal, as it requires a careful application-dependent selection of the types of augmentations to be performed, thus potentially biasing the information learned by the network during self-training. Moreover, several unsupervised methods only focus on uni-modal information, thus potentially introducing challenges in the case of sparse and textureless point clouds. To address these issues, we propose an augmentation-free unsupervised approach for point clouds, named CluRender, to learn transferable point-level features by leveraging uni-modal information for soft clustering and cross-modal information for neural rendering. Soft clustering enables self-training through a pseudo-label prediction task, where the affiliation of points to their clusters is used as a proxy under the constraint that these pseudo-labels divide the point cloud into approximate equal partitions. This allows us to formulate a clustering loss to minimize the standard cross-entropy between pseudo and predicted labels. Neural rendering generates photorealistic renderings from various viewpoints to transfer photometric cues from 2D images to the features. The consistency between rendered and real images is then measured to form a fitting loss, combined with the cross-entropy loss to self-train networks. Experiments on downstream applications, including 3D object detection, semantic segmentation, classification, part segmentation, and few-shot learning, demonstrate the effectiveness of our framework in outperforming state-of-the-art techniques.

Blender as an Alternative to Architectural Apps

Blender, a 3D modeling and animation software, holds significant potential as a vital tool in architecture due to its varied capabilities. This abstract elaborates on the relevance of Blender in architectural practice and research. The paper highlights Blender's capacity to facilitate architectural visualization, design iteration, and virtual prototyping. It iterates on how Blender empowers architects to create immersive, photorealistic renderings, aiding in communication with clients and stakeholders. Moreover, the abstract delves into Blender's role in exploring innovative architectural concepts through parametric modeling, simulation, and experimentation. Keywords: 3D Modeling, Animation, Rendering, Texturing, Rigging, UV Mapping

An Examination of the Relationship Between Visualization Media and Consumer Product Evaluation.

Virtual product presentations that rely on static images and text are often insufficient to communicate all the information that is necessary to accurately evaluate a product. Technologies such as Virtual Reality (VR) or Augmented Reality (AR) have enabled more sophisticated representation methods, but certain product characteristics are difficult to assess and may result in perceptual differences when a product is evaluated in different visual media. In this article, we report two case studies in which a group of participants evaluated three designs of two product typologies (i.e., a desktop telephone and a coffee maker) as presented in three different visual media (i.e., photorealistic renderings, AR, and VR for the first case study; and photographs, a non-immersive virtual environment, and AR for the second case study) using eight semantic scales. An inferential statistical method using Aligned Rank Transform (ART) proceedings was applied to determine perceptual differences between groups. Our results show that in both cases product attributes in Jordan's physio-pleasure category are the most affected by the presentation media. The socio-pleasure category was also affected for the case of the coffee makers. The level of immersion afforded by the medium significantly affects product evaluation.

Transforming Sketches into Realistic Images: Leveraging Machine Learning and Image Processing for Enhanced Architectural Visualization

This article presents a novel approach for transforming architectural sketches into realistic images through the utilization of machine learning and image processing techniques. The proposed method leverages the Stable Diffusion model, a deep learning framework specifically designed for text-to-image generation. By integrating image processing algorithms into the workflow, the model gains a better understanding of the input sketches, resulting in visually coherent and meaningful output images. The study explores the application of the Stable Diffusion model in the context of architectural design, showcasing its potential to enhance the visualization process and support designers in generating accurate and compelling representations. The efficacy of the method is evaluated through qualitative assessment, demonstrating its effectiveness in bridging the gap between initial sketches and photorealistic renderings. This research contributes to the growing body of knowledge on the integration of machine learning and image processing in architecture, providing insights and practical implications for architects, design professionals and researchers in the field.

PDRF: Progressively Deblurring Radiance Field for Fast Scene Reconstruction from Blurry Images

We present Progressively Deblurring Radiance Field (PDRF), a novel approach to efficiently reconstruct high quality radiance fields from blurry images. While current State-of-The-Art (SoTA) scene reconstruction methods achieve photo-realistic renderings from clean source views, their performances suffer when the source views are affected by blur, which is commonly observed in the wild. Previous deblurring methods either do not account for 3D geometry, or are computationally intense. To addresses these issues, PDRF uses a progressively deblurring scheme for radiance field modeling, which can accurately model blur with 3D scene context. PDRF further uses an efficient importance sampling scheme that results in fast scene optimization. We perform extensive experiments and show that PDRF is 15X faster than previous SoTA while achieving better performance on both synthetic and real scenes.

3D Digital Preservation, Presentation, and Interpretation of Wooden Cultural Heritage on the Example of Sculptures of the FormaViva Kostanjevica Na Krki Collection

The paper presents an interdisciplinary approach to the treatment of the FormaViva collection of wooden sculptures exhibited outdoors in a natural environment near the Božidar Jakac Art Museum in Kostanjevica na Krki in Slovenia. The study focuses on 3D graphic representations of sculptures created with photogrammetry and 3D modelling. The results are photorealistic renderings, interactive presentations, 3D printed reproductions, jewellery, and interpretive animations. The research results show that graphic documentation techniques on 3D models allow for a more detailed investigation of the original structural identity of the sculpture. By incorporating 3D and interactive technologies, we are expanding the usability of cultural heritage objects. By using interpretive techniques that have led to jewellery and interpretive animations in our research, we not only breathe new life into the sculptures, but also enrich the stories of the sculptures with our own experiences of the sculptural work.

FreeStyleGAN

Current Generative Adversarial Networks (GANs) produce photorealistic renderings of portrait images. Embedding real images into the latent space of such models enables high-level image editing. While recent methods provide considerable semantic control over the (re-)generated images, they can only generate a limited set of viewpoints and cannot explicitly control the camera. Such 3D camera control is required for 3D virtual and mixed reality applications. In our solution, we use a few images of a face to perform 3D reconstruction, and we introduce the notion of the GAN camera manifold, the key element allowing us to precisely define the range of images that the GAN can reproduce in a stable manner. We train a small face-specific neural implicit representation network to map a captured face to this manifold and complement it with a warping scheme to obtain free-viewpoint novel-view synthesis. We show how our approach - due to its precise camera control - enables the integration of a pre-trained StyleGAN into standard 3D rendering pipelines, allowing e.g., stereo rendering or consistent insertion of faces in synthetic 3D environments. Our solution proposes the first truly free-viewpoint rendering of realistic faces at interactive rates, using only a small number of casual photos as input, while simultaneously allowing semantic editing capabilities, such as facial expression or lighting changes.

Live speech portraits

To the best of our knowledge, we first present a live system that generates personalized photorealistic talking-head animation only driven by audio signals at over 30 fps. Our system contains three stages. The first stage is a deep neural network that extracts deep audio features along with a manifold projection to project the features to the target person's speech space. In the second stage, we learn facial dynamics and motions from the projected audio features. The predicted motions include head poses and upper body motions, where the former is generated by an autoregressive probabilistic model which models the head pose distribution of the target person. Upper body motions are deduced from head poses. In the final stage, we generate conditional feature maps from previous predictions and send them with a candidate image set to an image-to-image translation network to synthesize photorealistic renderings. Our method generalizes well to wild audio and successfully synthesizes high-fidelity personalized facial details, e.g., wrinkles, teeth. Our method also allows explicit control of head poses. Extensive qualitative and quantitative evaluations, along with user studies, demonstrate the superiority of our method over state-of-the-art techniques.

Review: The New Farm: Contemporary Rural Architecture

Review: <i>The New Farm: Contemporary Rural Architecture</i>

Visual Navigation Among Humans With Optimal Control as a Supervisor

Real world visual navigation requires robots to operate in unfamiliar, human-occupied dynamic environments. Navigation around humans is especially difficult because it requires anticipating their future motion, which can be quite challenging. We propose an approach that combines learning-based perception with model-based optimal control to navigate among humans based only on monocular, first-person RGB images. Our approach is enabled by our novel data-generation tool, HumANav, that allows for photorealistic renderings of indoor environment scenes with humans in them, which are then used to train the perception module entirely in simulation. Through simulations and experiments on a mobile robot, we demonstrate that the learned navigation policies can anticipate and react to humans without explicitly predicting future human motion, generalize to previously unseen environments and human behaviors, and transfer directly from simulation to reality. Videos describing our approach and experiments, as well as a demo of HumANav are available on the project website. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Deep relightable textures

The increasing demand for 3D content in augmented and virtual reality has motivated the development of volumetric performance capture systemsnsuch as the Light Stage. Recent advances are pushing free viewpoint relightable videos of dynamic human performances closer to photorealistic quality. However, despite significant efforts, these sophisticated systems are limited by reconstruction and rendering algorithms which do not fully model complex 3D structures and higher order light transport effects such as global illumination and sub-surface scattering. In this paper, we propose a system that combines traditional geometric pipelines with a neural rendering scheme to generate photorealistic renderings of dynamic performances under desired viewpoint and lighting. Our system leverages deep neural networks that model the classical rendering process to learn implicit features that represent the view-dependent appearance of the subject independent of the geometry layout, allowing for generalization to unseen subject poses and even novel subject identity. Detailed experiments and comparisons demonstrate the efficacy and versatility of our method to generate high-quality results, significantly outperforming the existing state-of-the-art solutions.

Deferred neural rendering

The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering , a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures , which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.

Physically principled reflection models applied to filtered camera imaging inversions in metal walled fusion machines.

Ray-tracing techniques are applied to filtered divertor imaging, a diagnostic that has long suffered from artifacts due to the polluting effect of reflected light in metal walled fusion machines. Physically realistic surface reflections were modeled using a Cook-Torrance micro-facet bi-directional reflection distribution function applied to a high resolution mesh of the vessel geometry. In the absence of gonioreflectometer measurements, a technique was developed to fit the free parameters of the Cook-Torrance model against images of the JET in-vessel light sources. By coupling this model with high fidelity plasma fluid simulations, photo-realistic renderings of a number of tokamak plasma emission scenarios were generated. Finally, a sensitivity matrix describing the optical coupling of a JET divertor camera and the emission profile of the plasma was obtained, including full reflection effects. These matrices are used to perform inversions on measured data and shown to reduce the level of artifacts in inverted emission profiles.