Photorealistic Images Research Articles

SAW-GAN: Multi-granularity Text Fusion Generative Adversarial Networks for text-to-image generation

Text-to-image generation is a challenging task that aims to generate visually realistic images semantically consistent for a given text. Existing methods mainly exploit the global semantic information of a single sentence while ignoring fine-grained semantic information such as aspects and words, which are critical factors in bridging the semantic gap in text-to-image generation. We propose a Multi-granularity Text (Sentence-level, Aspect-level, and Word-level) Fusion Generative Adversarial Network (SAW-GAN), which comprehensively represents textual information from multiple granularities. To effectively fuse multi-granularity information, we design a Double-granularity-text Fusion Module (DFM) fusing sentence and aspect information through parallel affine transformation and a Triple granularity-text Fusion Module (TFM) fusing sentence, aspect and word information by designing a novel Coordinate Attention Module (CAM), which can precisely locate the visual areas associated with each aspect and word. Furthermore, we use CLIP (Contrastive Language-Image Pre-training) to provide visual information to bridge the semantic gap and improve the model’s generalization ability. Our results show significant performance improvements over state-of-the-art methods using Conditional Generation Adversarial Network (CGAN) on CUB (FID from 13.91 to 10.45) and COCO (FID from 14.60 to 11.17) datasets with photorealistic images of richer details and text–image consistency.

High Space‐Bandwidth‐Product (SBP) Hologram Carriers Toward Photorealistic 3D Holography

Abstract3D holography capable of reproducing all necessary visual cues is considered the most promising route to present photorealistic 3D images. Three elements involving computer‐generated hologram (CGH) algorithms, hologram carriers, and optical systems are prerequisites to create high‐quality holographic displays for photorealistic 3D holography. Especially, the hologram carrier directly determines the holographic display capability and the design of high space‐bandwidth‐product (SBP) optical systems. Currently, two categories of hologram carriers, i.e., spatial light modulators (SLM) and metasurfaces, are regarded as promising candidates for photorealistic 3D holography. However, most of their SBP capability still cannot match the amount of information generated by the CGH. To address this issue, tremendous efforts are made to improve the capability of hologram carriers. Here, the main hologram carriers (from SLM to metasurfaces) that are widely utilized in holography systems to achieve high SBP capability (high resolution, wide viewing angles, and large sizes) are reviewed. The purpose of this review is to identify the key challenges and future directions of SLM‐based and metasurface‐based holography for photorealistic 3D holographic images.

Comparative analysis of approaches necessary for designing a robotic module for industrial parts defect detection

Over the past two decades, the most frequently implemented method of oil extraction in the Russian Federation has been a mechanized method using an electric submersible pump (ESP) unit. Due to the remoteness of the fields from the base of repair enterprises, the cost of transporting spent but maintainable ESPs significantly exceeds their purchase price, which over time leads to the formation of a significant amount of decommissioned oil production equipment. To solve this problem, it is proposed to develop a mobile robotic module for sorting, defecting and storing pump parts, thanks to its use, minor repairs of equipment will be possible directly at production sites. The article deals with the problems of complex flaw detection of metal and non-metallic parts with the possibility of applying the developed methodology for a wide range of industrial products, at the same time, the main attention is paid to work with ESP parts. Based on the decomposition of the problem, the most problematic operations were identified: classification of parts, surface control (identification of defects), dimensional control. The results of a brief comparative analysis for each of the above subtasks are based on a review of the scientific literature over the past 30 years, with the largest number of sources reviewed in the last 5 years. As a result optimal methods for solving the task were derived — a machine learning technique for classifying surface defects, the use of a coordinate measuring machine with a manipulator for dimensional control. A new approach is also proposed to solve the main problem of machine learning methods (lack of training samples) in the form of using synthetic photorealistic images for classification with transfer of defect features from semantically close and publicly available training samples.

DermSynth3D: Synthesis of in-the-wild annotated dermatology images

In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermatological images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.

AltDiffusion: A Multilingual Text-to-Image Diffusion Model

Large Text-to-Image(T2I) diffusion models have shown a remarkable capability to produce photorealistic and diverse images based on text inputs. However, existing works only support limited language input, e.g., English, Chinese, and Japanese, leaving users beyond these languages underserved and blocking the global expansion of T2I models. Therefore, this paper presents AltDiffusion, a novel multilingual T2I diffusion model that supports eighteen different languages. Specifically, we first train a multilingual text encoder based on the knowledge distillation. Then we plug it into a pretrained English-only diffusion model and train the model with a two-stage schema to enhance the multilingual capability, including concept alignment and quality improvement stage on a large-scale multilingual dataset. Furthermore, we introduce a new benchmark, which includes Multilingual-General-18(MG-18) and Multilingual-Cultural-18(MC-18) datasets, to evaluate the capabilities of T2I diffusion models for generating high-quality images and capturing culture-specific concepts in different languages. Experimental results on both MG-18 and MC-18 demonstrate that AltDiffusion outperforms current state-of-the-art T2I models, e.g., Stable Diffusion in multilingual understanding, especially with respect to culture-specific concepts, while still having comparable capability for generating high-quality images. All source code and checkpoints could be found in https://github.com/superhero-7/AltDiffuson.

MagiCapture: High-Resolution Multi-Concept Portrait Customization

Large-scale text-to-image models including Stable Diffusion are capable of generating high-fidelity photorealistic portrait images. There is an active research area dedicated to personalizing these models, aiming to synthesize specific subjects or styles using provided sets of reference images. However, despite the plausible results from these personalization methods, they tend to produce images that often fall short of realism and are not yet on a commercially viable level. This is particularly noticeable in portrait image generation, where any unnatural artifact in human faces is easily discernible due to our inherent human bias. To address this, we introduce MagiCapture, a personalization method for integrating subject and style concepts to generate high-resolution portrait images using just a few subject and style references. For instance, given a handful of random selfies, our fine-tuned model can generate high-quality portrait images in specific styles, such as passport or profile photos. The main challenge with this task is the absence of ground truth for the composed concepts, leading to a reduction in the quality of the final output and an identity shift of the source subject. To address these issues, we present a novel Attention Refocusing loss coupled with auxiliary priors, both of which facilitate robust learning within this weakly supervised learning setting. Our pipeline also includes additional post-processing steps to ensure the creation of highly realistic outputs. MagiCapture outperforms other baselines in both quantitative and qualitative evaluations and can also be generalized to other non-human objects.

Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

Reconstructing high-fidelity hand models with intricate textures plays a crucial role in enhancing human-object interaction and advancing real-world applications. Despite the state-of-the-art methods excelling in texture generation and image rendering, they often face challenges in accurately capturing geometric details. Learning-based approaches usually offer better robustness and faster inference, which tend to produce smoother results and require substantial amounts of training data. To address these issues, we present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details. Firstly, our approach predicts a parametric hand mesh model through Graph Convolutional Networks (GCN) based method from multi-view images. We further introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures, which is capable of preserving the mesh topology. In addition, we suggest an effective mesh-based neural rendering scheme to simultaneously generate photo-realistic image and optimize mesh geometry by fusing the pre-trained rendering network with vertex features. We conduct the comprehensive experiments on InterHand2.6M, DeepHandMesh and dataset collected by ourself, whose promising results show that our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality. Code and dataset are publicly available at https://github.com/agnJason/FMHR.

Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model

Recently, diffusion-based image generation methods are credited for their remarkable text-to-image generation capabilities, while still facing challenges in accurately generating multilingual scene text images. To tackle this problem, we propose Diff-Text, which is a training-free scene text generation framework for any language. Our model outputs a photo-realistic image given a text of any language along with a textual description of a scene. The model leverages rendered sketch images as priors, thus arousing the potential multilingual-generation ability of the pre-trained Stable Diffusion. Based on the observation from the influence of the cross-attention map on object placement in generated images, we propose a localized attention constraint into the cross-attention layer to address the unreasonable positioning problem of scene text. Additionally, we introduce contrastive image-level prompts to further refine the position of the textual region and achieve more accurate scene text generation. Experiments demonstrate that our method outperforms the existing method in both the accuracy of text recognition and the naturalness of foreground-background blending.

Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition

Recognition in open-world scenarios is an important and challenging field, where Vision-Language Pre-training paradigms have greatly impacted the 2D domain. This inspires a growing interest in introducing 2D pre-trained models, such as CLIP, into the 3D domain to enhance the ability of point cloud understanding. Considering the difference between discrete 3D point clouds and real-world 2D images, reducing the domain gap is crucial. Some recent works project point clouds onto a 2D plane to enable 3D zero-shot capabilities without training. However, this simplistic approach leads to an unclear or even distorted geometric structure, limiting the potential of 2D pre-trained models in 3D. To address the domain gap, we propose Point2Real, a training-free framework based on the realistic rendering technique to automate the transformation of the 3D point cloud domain into the Vision-Language domain. Specifically, Point2Real leverages a shape recovery module that devises an iterative ball-pivoting algorithm to convert point clouds into meshes, narrowing the gap in shape at first. To simulate photo-realistic images, a set of refined textures as candidates is applied for rendering, where the CLIP confidence is utilized to select the suitable one. Moreover, to tackle the viewpoint challenge, a heuristic multi-view adapter is implemented for feature aggregation, which exploits the depth surface as an effective indicator of view-specific discriminability for recognition. We conduct experiments on ModelNet10, ModelNet40, and ScanObjectNN datasets, and the results demonstrate that Point2Real outperforms other approaches in zero-shot and few-shot tasks by a large margin.

Relightable and Animatable Neural Avatars from Videos

Lightweight creation of 3D digital avatars is a highly desirable but challenging task. With only sparse videos of a person under unknown illumination, we propose a method to create relightable and animatable neural avatars, which can be used to synthesize photorealistic images of humans under novel viewpoints, body poses, and lighting. The key challenge here is to disentangle the geometry, material of the clothed body, and lighting, which becomes more difficult due to the complex geometry and shadow changes caused by body motions. To solve this ill-posed problem, we propose novel techniques to better model the geometry and shadow changes. For geometry change modeling, we propose an invertible deformation field, which helps to solve the inverse skinning problem and leads to better geometry quality. To model the spatial and temporal varying shading cues, we propose a pose-aware part-wise light visibility network to estimate light occlusion. Extensive experiments on synthetic and real datasets show that our approach reconstructs high-quality geometry and generates realistic shadows under different body poses. Code and data are available at https://wenbin-lin.github.io/RelightableAvatar-page.

Abstract 3799: Spatial biomarker prediction in papillary renal cell carcinoma using artificial intelligence-based immunofluorescent inference

Abstract Papillary renal cell carcinoma (pRCC) accounts for ~15-20% of the global burden for kidney cancers but still lacks effective mechanisms to predict patient responses to therapeutic intervention. Existing therapies targeting kidney cancers (anti-angiogenesis and immunotherapy) intimately involve the tumor microenvironment in their mechanism of action. However, the role of the physical organization of the tumor in determining therapeutic efficacy remains unknown. Previous efforts have utilized transcriptional profiling to stratify patients into treatment categories but here we propose to leverage artificial neural networks (“AI”) to infer the expression of marker genes for particular cell types of interest based on ubiquitously expressed immunofluorescence channels, unstained brightfield images, and hematoxylin and eosin (H&E) stained images. We have developed AI models that are capable of the prediction of a range of relevant cell types based on the expression profile of DAPI, type IV collagen, and integrin beta-1, including endothelial cells, T cells, macrophages and other cell types. This work applies an AI model class known as generative adversarial networks (GANs) to produce photo-realistic simulated immunofluorescence images based on the input channels. This approach may allow us to accurately predict complex multiplex immunofluorescence (mIF) images from simpler, more routine histological methods, greatly reducing the cost burden required to profile histopathology images for cell types of interest. Our previous work identified a number of interesting cell-cell association clusters correlated with patient prognosis and this extension of that previous work aims to allow for the prediction of these interactions while obviating the need to perform the original, costly multiplex immunofluorescence assay or even a more reduced immunofluorescence staining panel which may still be out of reach for many clinical pathology labs and the interpretation of which may be challenging. By simplifying and increasing the availability of predictive AI models for digital pathology, we hope to further develop and apply predictive biomarkers for patient prognosis and treatment response in papillary renal cell carcinoma. Citation Format: Jared Brewer, Joseph Vento, Scott Haake, Anupama Reddy. Spatial biomarker prediction in papillary renal cell carcinoma using artificial intelligence-based immunofluorescent inference [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 3799.

Cinematic Rendering of Pancreatic Neuroendocrine Tumours: Opportunities for Clinical Implementation: Part 2: Preoperative Planning and Evaluation of Metastatic Disease.

Pancreatic neuroendocrine tumours (PNETs) are a rare subset of pancreatic tumours that have historically comprised up to 3% of all clinically detected pancreatic tumours. In recent decades, however, advancements in imaging have led to an increased incidental detection rate of PNETs and imaging has played an increasingly central role in the initial diagnostics and surgical planning of these tumours. Cinematic rendering (CR) is a 3D post-processing technique that generates highly photorealistic images through more realistically modelling the path of photons through the imaged volume. This allows for more comprehensive visualization, description, and interpretation of anatomical structures. In this 2-part review article, we present the first description of the various CR appearances of PNETs in the reported literature while providing commentary on the unique clinical opportunities afforded by the adjunctive utilization of CR in the workup of these rare tumours. This second instalment focuses on the applications of CR in optimizing preoperative planning of PNETs.

Cinematic Rendering of Pancreatic Neuroendocrine Tumours: Opportunities for Clinical Implementation: Part 1: Tumour Detection and Characterization.

Pancreatic neuroendocrine tumours (PNETs) are a rare subset of pancreatic tumours that have historically comprised up to 3% of all clinically detected pancreatic tumours. In recent decades, however, advancements in imaging have led to an increased incidental detection rate of PNETs and imaging has played an increasingly central role in the initial diagnostics and surgical planning of these tumours. Cinematic rendering (CR) is a 3D post-processing technique that generates highly photorealistic images through more realistically modelling the path of photons through the imaged volume. This allows for more comprehensive visualization, description, and interpretation of anatomical structures. In this 2-part review article, we present the first description of the various CR appearances of PNETs in the reported literature while providing commentary on the unique clinical opportunities afforded by the adjunctive utilization of CR in the workup of these rare tumours. The first of these 2 instalments highlights the utility of CR in optimizing PNET detection and characterization.

Photorealistic style transfer fusing frequency separation channel attention mechanism and mirror flow pyramid integration

Photorealistic image style transfer aims to transfer style information while preserving the realistic details of the content image. However, an existing limitation is the inability to effectively balance the relationship between image realism and stylization intensity, resulting in poor image transfer performance. To address this issue, we propose an photorealistic style transfer method that fusing Frequency Separation Channel Attention Mechanism (FSCAM) and Mirror Fluid Pyramid Integration (MFPI). This method achieves superior stylization intensity while improves image realism. Firstly, we propose an improved channel attention mechanism called FSCAM. This mechanism utilizes Discrete Cosine Transform (DCT) to decompose features into different frequency components and screens out high-valued texture and color features, thereby enhancing the stylization intensity of the generated images. In addition, we designed a MFPI module. The module is able to integrate information from different scales, enhance the preservation of low-level detail features in high-level features, and thus improve the realism of the images. Experimental results demonstrate that our method not only enhances the stylization intensity but also improves the image realism. It achieves satisfactory performance in terms of subjective visual performance and objective evaluation metrics.

Reimagining learning through AI art: the promise of DALL-E and MidJourney for education and libraries

Purpose This paper aims to explore the potential impact of artificial intelligence (AI) image generators, specifically MidJourney and DALL-E, on education and library services. The study aims to understand how these tools can revolutionize learning experiences and library resources while also addressing the ethical considerations surrounding their use. Design/methodology/approach This study investigates the technical foundations of MidJourney and DALL-E, highlighting their neural network architectures. It also traces the iterative refinement of these models and examines cost, accessibility and the unique prompt-guided capabilities of DALL-E 3. Findings MidJourney and DALL-E show remarkable progress in generating high-quality, photorealistic images from text prompts. The iterative refinement of these models demonstrates a trend toward improved creative output and user accessibility. DALL-E 3, in particular, allows users to guide image generation through prompt modifications, offering unprecedented control over the creative process. The study identifies potential applications in personalized learning, visual communication and research support in libraries, while recognizing challenges such as cost and accessibility. Originality/value This research innovatively explores AI's impact on education and libraries, detailing applications in personalized learning and research while addressing legal and ethical considerations.

Generative adversarial networks with attentional multimodal for human face synthesis

<p><span>Face synthesis and editing has increased cumulative consideration by the improvement of generative adversarial networks (GANs). The proposed attentional GAN-deep attentional multimodal similarity modal (AttnGAN-DAMSM) model focus on generating high-resolution images by removing discriminator components and generating realistic images from textual description. The attention model creates the attention map on the image and automatically retrieves the features to produce various sub-areas of the image. The DAMSM delivers fine-grained image-text identical loss to generative networks. This study, first describe text phrases and the model will generate a photorealistic high-resolution image composed of features with high accuracy. Next, model will fine-tune the selected features of face images and it will be left to the control of the user. The result shows that the proposed AttnGAN-DAMSM model delivers the performance metrics like structural similarity index measure (SSIM), feature similarity index measure (FSIM) and frechet inception distance (FID) using CelebA and CUHK face sketch (CUFS) dataset. For CelebFaces attribute (CelebA) dataset, the SSIM achieves 78.82% and for CUFS dataset, the SSIM achieves 81.45% which ensures accurate face synthesis and editing compared with existing methods such as GAN, SuperstarGAN and identity-sensitive GAN (IsGAN) models.</span></p>

Variation-aware semantic image synthesis

Semantic image synthesis (SIS) aims to produce photorealistic images aligning to given conditional semantic layout and has witnessed a significant improvement in recent years. Although the diversity in image-level has been discussed heavily, class-level mode collapse widely exists in current algorithms. Therefore, we declare a new requirement for SIS to achieve more photorealistic images, variation-aware, which consists of inter- and intra-class variation. The inter-class variation is the diversity between different semantic classes while the intra-class variation stresses the diversity inside one class. Through analysis, we find that current algorithms elusively embrace the inter-class variation but the intra-class variation is still not enough. Further, we introduce two simple methods to achieve variation-aware semantic image synthesis (VASIS) with a higher intra-class variation, semantic noise and position code. We combine our method with several state-of-the-art algorithms and the experimental result shows that our models generate more natural images and achieve slightly better FIDs and/or mIoUs than the counterparts. Our codes and models will be publicly available.

GAN with opposition-based blocks and channel self-attention mechanism for image synthesis

Recently, image synthesis has always been a research hotspot in the field of deep learning. Generally, the methods based on generative adversarial networks (GANs) directly feed the semantic layout as input to obtain the photorealistic images for image synthesis. However, these methods based on GANs have not achieved satisfactory reconstructed results in quality. One of the main reasons is that the normalization layers in these methods will cause the loss of the semantic information. Another of the main reason is that the information contained in the semantic layout is sparse. In order to solve the above problems, GAN with opposition-based blocks and channel self-attention mechanism (OCGAN) is proposed. In OCGAN, the opposition-based learning method and the proposed adaptive normalization method are used to design the opposition-based blocks (OB Blks). The proposed channel self-attention mechanism (CSAM) is employed to give different focus to each channel of the semantic layout. The generator of OCGAN uses the opposition-based blocks and the channel self-attention mechanism to maintain and capture the important details from the semantic layouts. Experiments on several challenging datasets demonstrate the advantages of our method over existing approaches, regarding both visual quality and the representative evaluating criteria.

Advanced image generation for cancer using diffusion models.

Deep neural networks have significantly advanced the field of medical image analysis, yet their full potential is often limited by relatively small dataset sizes. Generative modeling, particularly through diffusion models, has unlocked remarkable capabilities in synthesizing photorealistic images, thereby broadening the scope of their application in medical imaging. This study specifically investigates the use of diffusion models to generate high-quality brain MRI scans, including those depicting low-grade gliomas, as well as contrast-enhanced spectral mammography (CESM) and chest and lung X-ray images. By leveraging the DreamBooth platform, we have successfully trained stable diffusion models utilizing text prompts alongside class and instance images to generate diverse medical images. This approach not only preserves patient anonymity but also substantially mitigates the risk of patient re-identification during data exchange for research purposes. To evaluate the quality of our synthesized images, we used the Fréchet inception distance metric, demonstrating high fidelity between the synthesized and real images. Our application of diffusion models effectively captures oncology-specific attributes across different imaging modalities, establishing a robust framework that integrates artificial intelligence in the generation of oncological medical imagery.

The role of artificial intelligence in generating original scientific research

Artificial intelligence (AI) is a revolutionary technology that is finding wide application across numerous sectors. Large language models (LLMs) are an emerging subset technology of AI and have been developed to communicate using human languages. At their core, LLMs are trained with vast amounts of information extracted from the internet, including text and images. Their ability to create human-like, expert text in almost any subject means they are increasingly being used as an aid to presentation, particularly in scientific writing. However, we wondered whether LLMs could go further, generating original scientific research and preparing the results for publication. We taskedGPT-4, an LLM, to write an original pharmaceutics manuscript, on a topic that is itself novel. It was able to conceive a research hypothesis, define an experimental protocol, produce photo-realistic images of 3D printed tablets, generate believable analytical data from a range of instruments and write a convincing publication-ready manuscript with evidence of critical interpretation. The model achieved all this is less than 1 h. Moreover, the generated data were multi-modal in nature, including thermal analyses, vibrational spectroscopy and dissolution testing, demonstrating multi-disciplinary expertise in the LLM. One area in which the model failed, however, was in referencing to the literature. Since the generated experimental results appeared believable though, we suggest that LLMs could certainly play a role in scientific research but with human input, interpretation and data validation. We discuss the potential benefits and current bottlenecks for realising this ambition here.