Image Generation Model Research Articles

Pseudo dual-energy CT-derived iodine mapping using single-energy CT data based on a convolution neural network.

The objectives of this study are: (1) to develop a convolutional neural network model that yields pseudo high-energy CT (CTpseudo_high) from simple image processed low-energy CT (CTlow) images, and (2) to create a pseudo iodine map (IMpseudo) and pseudo virtual non-contrast (VNCpseudo) images for thoracic and abdominal regions. Eighty patients who underwent dual-energy CT (DECT) examinations were enrolled. The data obtained from 55, 5, and 20 patients were used for training, validation, and testing, respectively. The ResUnet model was used for image generation model and was trained using CTlow and high-energy CT (CThigh) images. The proposed model performance was evaluated by calculating the CT values, image noise, mean absolute errors (MAEs), and histogram intersections (HIs). The mean difference in the CT values between CTpseudo_high and CThigh images were less than 6 Hounsfield unit (HU) for all evaluating patients. The image noise of CTpseudo_high was significantly lower than that of CThigh. The mean MAEs was less than 15 HU, and HIs were almost 1.000 for all the patients. The evaluation metrics of IM and VNC exhibited the same tendency as that of the comparison between CTpseudo_high and CThigh images. Our results indicated that the proposed model enables to obtain the DECT images and material-specific images from only single-energy CT images. We constructed the CNN-based model which can generate pseudo DECT image and DECT-derived material-specific image using only simple image-processed CTlow images for the thoracic and abdominal regions.

Qualitative failures of image generation models and their application in detecting deepfakes

The remarkable advancement of image and video generation models has led to the creation of exceptionally realistic content, posing challenges in differentiating between genuine and fabricated instances in numerous scenarios. However, despite this progress, a gap remains between the quality of generated images and those found in the real world. To address this, we have reviewed a vast body of literature from both academic publications and social media to identify qualitative shortcomings in image generation models, which we have classified into five categories. By understanding these failures, we can identify areas where these models need improvement, as well as develop strategies for detecting generated images and deepfakes. The prevalence of deepfakes in today’s society is a serious concern, and our findings can help mitigate their negative impact. In order to support research in this field, a collection of instances where models have failed is made available at here.

UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image

Text-driven image generation methods have shown impressive results recently, allowing casual users to generate high quality images by providing textual descriptions. However, similar capabilities for editing existing images are still out of reach. Text-driven image editing methods usually need edit masks, struggle with edits that require significant visual changes and cannot easily keep specific details of the edited portion. In this paper we make the observation that image-generation models can be converted to image-editing models simply by fine-tuning them on a single image. We also show that initializing the stochastic sampler with a noised version of the base image before the sampling and interpolating relevant details from the base image after sampling further increase the quality of the edit operation. Combining these observations, we propose UniTune, a novel image editing method. UniTune gets as input an arbitrary image and a textual edit description, and carries out the edit while maintaining high fidelity to the input image. UniTune does not require additional inputs, like masks or sketches, and can perform multiple edits on the same image without retraining. We test our method using the Imagen model in a range of different use cases. We demonstrate that it is broadly applicable and can perform a surprisingly wide range of expressive editing operations, including those requiring significant visual changes that were previously impossible.

FundusGAN: Fundus image synthesis based on semi-supervised learning

Our goal is to construct a high-performance model that generates two types of fundus disease images for both Diabetic Retinopathy (DR) and Age-Related Macular degeneration (AMD) that have different symptoms for each severity for overcoming the lack of labeled data and data imbalance problems that hinder to build a model based on deep learning. To this end, we propose a framework named FundusGAN for building fundus image generators in semi-supervised learning. First, a semantic coarse lesion mask as a guidance mask is exploited to express disease and severity information more accurately to generate a final fundus image by adopting a coarse-to-fine strategy. Second, the disease-feature matching loss is proposed to learn the abundant features of the two diseases by employing the unlabeled datasets and is able to generate the initial symptoms of two fundus diseases accurately. We demonstrate the effectiveness of the proposed FundusGAN in comparison with carefully tuned baselines and state-of-the-art DR fundus image generation model and finally achieve 0.9174, and 0.8254 f1-score for DR and AMD, respectively.

MVFSIGM: multi-variant feature-based synthesis image generation model for improved stability using generative adversarial network

MVFSIGM: multi-variant feature-based synthesis image generation model for improved stability using generative adversarial network

Reversible Modal Conversion Model for Thermal Infrared Tracking

Learning powerful CNN representation of the target is a key issue for thermal infrared (TIR) tracking. The lack of massive training TIR data is one of the obstacles to training the network in a end-to-end way from the scratch. Compared to the time-consuming and labor-intensive method of heavily relabeling data, we obtain trainable TIR images by leveraging the massive annotated RGB images in this paper. Unlike the traditional image generation models, a modal reversible module is designed to maximize the information propagation between RGB and TIR modals in this work. The advantage is that this module can preserve the modal information as possible when the network is conducted on a large number of aligned RGBT image pairs. Additionally, the fake-TIR features generated by the proposed module are also integrated to enhance the target representation ability when TIR tracking is on-the-fly. To verify the proposed method, we conduct sufficient experiments on both single-modal TIR and multi-modal RGBT tracking datasets. In single-modal TIR tracking, the performance of our method is improved by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2.8\%$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$0.94\%$</tex-math></inline-formula> on success rate compared with the SOTA on LSOTB-TIR and PTB-TIR dataset. In multi-modal RGBT fusion tracking, the proposed method is tested on the RGBT234 and VOT-RGBT2020 datasets and the results have also reached the performance of SOTA.

Inflating 2D convolution weights for efficient generation of 3D medical images

Background and Objective: The generation of three-dimensional (3D) medical images has great application potential since it takes into account the 3D anatomical structure. Two problems prevent effective training of a 3D medical generative model: (1) 3D medical images are expensive to acquire and annotate, resulting in an insufficient number of training images, and (2) a large number of parameters are involved in 3D convolution. Methods: We propose a novel GAN model called 3D Split&Shuffle-GAN. To address the 3D data scarcity issue, we first pre-train a two-dimensional (2D) GAN model using abundant image slices and inflate the 2D convolution weights to improve the initialization of the 3D GAN. Novel 3D network architectures are proposed for both the generator and discriminator of the GAN model to significantly reduce the number of parameters while maintaining the quality of image generation. Several weight inflation strategies and parameter-efficient 3D architectures are investigated. Results: Experiments on both heart (Stanford AIMI Coronary Calcium) and brain (Alzheimer’s Disease Neuroimaging Initiative) datasets show that our method leads to improved 3D image generation quality (14.7 improvements on Frchet inception distance) with significantly fewer parameters (only 48.5% of the baseline method). Conclusions: We built a parameter-efficient 3D medical image generation model. Due to the efficiency and effectiveness, it has the potential to generate high-quality 3D brain and heart images for real use cases.

Thinking Geographically about AI Sustainability

Abstract. Driven by foundation models, recent progress in AI and machine learning has reached unprecedented complexity. For instance, the GPT-3 language model consists of 175 billion parameters and a training-data size of 570 GB. While it has achieved remarkable performance in generating text that is difficult to distinguish from human-authored content, a single training of the model is estimated to produce over 550 metric tons of CO2 emissions. Likewise, we see advances in GeoAI research improving large-scale prediction tasks like satellite image classification and global climate modeling, to name but a couple. While these models have not yet reached comparable complexity and emissions levels, spatio-temporal models differ from language and image-generation models in several ways that make it necessary to (re)train them more often, with potentially large implications for sustainability. While recent work in the machine learning community has started calling for greener and more energy-efficient AI alongside improvements in model accuracy, this trend has not yet reached the GeoAI community at large. In this work, we bring this issue to not only the attention of the GeoAI community but also present ethical considerations from a geographic perspective that are missing from the broader, ongoing AI-sustainability discussion. To start this discussion, we propose a framework to evaluate models from several sustainability-related angles, including energy efficiency, carbon intensity, transparency, and social implications. We encourage future AI/GeoAI work to acknowledge its environmental impact as a step towards a more resource-conscious society. Similar to the current push for reproducibility, future publications should also report the energy/carbon costs of improvements over prior work.

Fully synthetic training for image restoration tasks

In this work, we show that neural networks aimed at solving various image restoration tasks can be successfully trained on fully synthetic data. In order to do so, we rely on a generative model of images, the scaling dead leaves model, which is obtained by superimposing disks whose size distribution is scale-invariant. Pairs of clean and corrupted synthetic images can then be obtained by a careful simulation of the degradation process. We show on various restoration tasks that such a synthetic training yields results that are only slightly inferior to those obtained when the training is performed on large natural image databases. This implies that, for restoration tasks, the geometric contents of natural images can be nailed down to only a simple generative model and a few parameters. This prior can then be used to train neural networks for specific modality, without having to rely on demanding campaigns of natural images acquisition. We demonstrate the feasibility of this approach on difficult restoration tasks, including the denoising of smartphone RAW images and the full development of low-light images.

Image generation models from scene graphs and layouts: A comparative analysis

An image is the abstraction of a thousand words. The meaning and essence of complex topics, ideas, and concepts can be easily and effectively conveyed visually by a single image rather than a lengthy verbal description. It is not only essential to teach computers how to recognize and classify images but also how to generate them. Controlled image generation depicting complex and multiple objects is a challenging task in computer vision despite the significant advancements in generative modeling. Among the core challenges, scene graph-based and scene layout-based image generation is a significant problem in computer vision and requires generative models to reason about object relationships and compositionality. Due to its ease of use, less time cost, and labor needs, image generation/synthesizing models from scene graphs and layouts are proliferating. In the case of a more significant number of scene graphs and layout to image generation models, a unique experimental evaluation methodology is required to evaluate the controlled image generation. To this extent, we, in this work, present a standard methodology to evaluate the performance of scene graph and scene layout-based image generation models. We perform a comparative analysis of image generation models to evaluate image generation models’ complexity from scene graphs and scene layouts. We analyze the different components of these models on Visual Genome and COCO-Stuff datasets. The experimental results show that the scene layout-based image generation outperforms its graph-based counterpart in most quantitative and qualitative evaluations.

How to Imagine the World with Text? From Text-to-image Generation View

Words are an effective and convenient way to describe the world, but sometimes what the texts convey may be misunderstood by readers. The expression of pictures is more vivid, easy to understand and has no borders, but creating a painting often takes a long time. Text-to-image makes the two expressions complement each other: It makes every ordinary person a “painter”, so that they can feel the world, express themselves, and create more whimsy through many rich pictures. For this vision, technologists are trying their best to improve image generation models, which enables computers to generate high quality images with texts better. And they are solving some technical defects, for instance, sometimes the content of generated images is strange. In the future, text-to-image can be adapted to applications in AI such as computer-aided design, image editing, and be employed in the field of art such as movies and artworks, and then it may even make a big difference on people's life, enriching the public's spiritual world and conveying information by vivid images.

P‐2.9: A review of image generation methods based on deep learning

Image as a medium of visual information transmission have the advantages of vividness, intuition and easy understanding. They play an important role in information transmission and utilization. In recent years, due to the rapid development of deep learning technology in the field of image processing, image generative model based on neural network has become one of the current research hotspots. In the field of deep learning, unsupervised learning model has received more and more attentions, especially in the field of deep generative models, which has made breakthrough progress [1] . Among them, Variational Auto-Encoder (VAE), Generative Adversarial Network (GAN), and Diffusion Model are the three most representative research methods in the field of unsupervised learning. They have been applied more and more in the field of deep generative models. Especially, the high-quality image generative models based on the generative adversarial network continue to be hot. The diffusion model is a rising star, which is favored by more and more researchers. This paper first summarizes the main research work, improvement mechanism and features of image generation methods based on VAE and GAN, then introduces the principle of the rising diffusion model and its representative models. Finally, the advantages and limitations of the above methods are compared and analyzed, and prospects for future research are put forward.

감각치환을 위한 소리 기반 감정 표현 얼굴 생성 연구

Research on Visual-to-Auditory (V2A) which uses neural facilitation to transfer visual information to auditory information is actively ongoing. However, existing V2A methods have provided users with limited information by simply converting visual information to auditory information. To provide more information to the users, it is necessary to generate and transmit augmented visual information by integrating it with other sensory information. Recently, methods for generating augmented visual information by integrating auditory and visual information are being studied, but there is a drawback in that they cannot generate images optimized for sound intensity by not considering the rich intensity of sound. To solve this problem, in this paper, a method of classifying sound and converting the classified result into sound intensity to generate images optimized for sound intensity is proposed. The results of applying the proposed method to a face image generation model in experiments showed that the proposed method can generate face images utilizing sound information when sound intensity is considered.

Biomaterials Research-Driven Design Visualized by AI Text-Prompt-Generated Images

AI text-to-image generated images have revolutionized the design process and its rapid development since 2022. Generating various iterations of perfect renders in few seconds by textually expressing the design concept. This high-potential tool has opened wide possibilities for biomaterials research-driven design. That is based on developing biomaterials for multi-scale applications in the design realm and built environment. From furniture to architectural elements to architecture. This approach to the design process has been augmented by the massive capacity of AI text-to-image models to visualize high-fidelity and innovative renders that reflect very detailed physical characteristics of the proposed biomaterials from micro to macro. However, this biomaterials research-driven design approach aided by AI text-to-image models requires criteria for evaluating the role and efficiency of employing AI image generation models in this design process. Furthermore, since biomaterials research-driven design is focused not only on design studies but also the biomaterials engineering research and process, it requires a sufficient method for protecting its novelty and copyrights. Since their emergence in late 2022, AI text-to-image models have been raising alarming ethical concerns about design authorship and designer copyrights. This requires the establishment of a referencing method to protect the copyrights of the designers of these generated renders as well as the copyrights of the authors of their training data referencing by proposing an auxiliary AI model for automatic referencing of these AI-generated images and their training data as well. Thus, the current work assesses the role of AI text-to-image models in the biomaterials research-driven design process and their methodology of operation by analyzing two case studies of biomaterials research-driven design projects performed by the authors aided by AI text-to-image models. Based on the results of this analysis, design criteria will be presented for a fair practice of AI-aided biomaterials research-driven process.

UAV Aerial Image Generation of Crucial Components of High-Voltage Transmission Lines Based on Multi-Level Generative Adversarial Network

With the aim of improving the image quality of the crucial components of transmission lines taken by unmanned aerial vehicles (UAV), a priori work on the defective fault location of high-voltage transmission lines has attracted great attention from researchers in the UAV field. In recent years, generative adversarial nets (GAN) have achieved good results in image generation tasks. However, the generation of high-resolution images with rich semantic details from complex backgrounds is still challenging. Therefore, we propose a novel GANs-based image generation model to be used for the critical components of power lines. However, to solve the problems related to image backgrounds in public data sets, considering that the image background of the common data set CPLID (Chinese Power Line Insulator Dataset) is simple. However, it cannot fully reflect the complex environments of transmission line images; therefore, we established an image data set named “KCIGD” (The Key Component Image Generation Dataset), which can be used for model training. CFM-GAN (GAN networks based on coarse–fine-grained generators and multiscale discriminators) can generate the images of the critical components of transmission lines with rich semantic details and high resolutions. CFM-GAN can provide high-quality image inputs for transmission line fault detection and line inspection models to guarantee the safe operation of power systems. Additionally, we can use these high-quality images to expand the data set. In addition, CFM-GAN consists of two generators and multiple discriminators, which can be flexibly applied to image generation tasks in other scenarios. We introduce a penalty mechanism-related Monte Carlo search (MCS) approach in the CFM-GAN model to introduce more semantic details in the generated images. Moreover, we presented a multiscale discriminator structure according to the multitask learning mechanisms to effectively enhance the quality of the generated images. Eventually, the experiments using the CFM-GAN model on the KCIGD dataset and the publicly available CPLID indicated that the model used in this work outperformed existing mainstream models in improving image resolution and quality.

주성분 분석을 잠재공간에 적용한 최소제곱 오차 이미지 생성 네트워크

Image generation is an important area of artificial intelligence that involves creating new images from existing dataset. It involves learning the distribution of target images from randomly generated vectors. Like other deep learning models, the image generation model requires a vast refined data set to produce high-quality results. When there is little data, there is a problem that the diversity and quality of generated images are compromised. In this paper, we propose a new generative model that applies PCA to the generator of the least square error adversarial generative network that, in turn, generates high-quality images even with a small data set. Unlike the existing models that generate target data from randomly generated noise, in the proposed method the direction of the image to be generated is guided by extracting the features of the target data through PCA. The results section shows the superior performance of the proposed model against a different number of images in dataset.

GH-DDM: the generalized hybrid denoising diffusion model for medical image generation

GH-DDM: the generalized hybrid denoising diffusion model for medical image generation

On Chatbots and Generative Artificial Intelligence.

On Chatbots and Generative Artificial Intelligence.

Unsupervised learning reveals interpretable latent representations for translucency perception.

Humans constantly assess the appearance of materials to plan actions, such as stepping on icy roads without slipping. Visual inference of materials is important but challenging because a given material can appear dramatically different in various scenes. This problem especially stands out for translucent materials, whose appearance strongly depends on lighting, geometry, and viewpoint. Despite this, humans can still distinguish between different materials, and it remains unsolved how to systematically discover visual features pertinent to material inference from natural images. Here, we develop an unsupervised style-based image generation model to identify perceptually relevant dimensions for translucent material appearances from photographs. We find our model, with its layer-wise latent representation, can synthesize images of diverse and realistic materials. Importantly, without supervision, human-understandable scene attributes, including the object's shape, material, and body color, spontaneously emerge in the model's layer-wise latent space in a scale-specific manner. By embedding an image into the learned latent space, we can manipulate specific layers' latent code to modify the appearance of the object in the image. Specifically, we find that manipulation on the early-layers (coarse spatial scale) transforms the object's shape, while manipulation on the later-layers (fine spatial scale) modifies its body color. The middle-layers of the latent space selectively encode translucency features and manipulation of such layers coherently modifies the translucency appearance, without changing the object's shape or body color. Moreover, we find the middle-layers of the latent space can successfully predict human translucency ratings, suggesting that translucent impressions are established in mid-to-low spatial scale features. This layer-wise latent representation allows us to systematically discover perceptually relevant image features for human translucency perception. Together, our findings reveal that learning the scale-specific statistical structure of natural images might be crucial for humans to efficiently represent material properties across contexts.

A New Perspective on Stabilizing GANs Training: Direct Adversarial Training

Generative Adversarial Networks (GANs) are the most popular image generation models that have achieved remarkable progress on various computer vision tasks. However, training instability is still one of the open problems for all GAN-based algorithms. Quite a number of methods have been proposed to stabilize the training of GANs, the focuses of which were respectively put on the loss functions, regularization and normalization technologies, training algorithms, and model architectures. Different from the above methods, in this paper, a new perspective on stabilizing GANs training is presented. It is found that sometimes the images produced by the generator act like adversarial examples of the discriminator during the training process, which may be part of the reason causing the unstable training of GANs. With this finding, we propose the Direct Adversarial Training (DAT) method to stabilize the training process of GANs. Furthermore, we prove that the DAT method is able to minimize the Lipschitz constant of the discriminator adaptively. The advanced performance of DAT is verified on multiple loss functions, network architectures, hyper-parameters, and datasets. Specifically, DAT achieves significant improvements of 11.5% FID on CIFAR-100 unconditional generation based on SSGAN, 10.5% FID on STL-10 unconditional generation based on SSGAN, and 13.2% FID on LSUN-Bedroom unconditional generation based on SSGAN. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><sup/></xref>