High-quality Image Generation Research Articles

RealtimeGen: An Intervenable AI Image Generation System for Commercial Digital Art Asset Creators

Recent advances in artificial intelligence-generated content (AIGC) have led to the rapid generation of high-quality images. AIGC has attracted the attention of commercial digital art asset creators. Traditional artist-led processes contrast with current AI tools that often reduce creators to passive roles. This study examines the integration of AI image generation into commercial digital art, emphasizing the importance of preserving creators’ creative autonomy. Our formative study (S1) involved interviews with commercial digital art creators, highlighting a need for greater control and transparency in AI-assisted painting. In response, we developed RealtimeGen, an integrated tool that merges human creativity with AI’s capabilities, allowing creators to intervene in the generative process. A user study (S2) comparing RealtimeGen with the popular AIGC tool Stable Diffusion was also carried out. The results showed its enhanced user experience and workflow compatibility. Our work contributes to understanding and improving AI-assisted painting workflows for commercial creators, offering them greater creative agency.

Color Transfer for Images: A Survey

High-quality image generation is an important topic in digital visualization. As a sub-topic of the research, color transfer is to produce a high-quality image with ideal color scheme learned from the reference one. In this article, we investigate the mainstream methods of color transfer to provide a survey that introduces the related theories and frameworks. Such methods can be divided into three categories: statistical color transfer, semantic-based color transfer, and color transfer for special target. For these mainstream technical routes, we discuss the related research background, technical details, and representative methods. We also exhibit some new trends of the topic according to recent progress. Based on the comparisons, we discuss the unsolved issues of color transfer and potential solutions in future work.

SIMFusion: A semantic information-guided modality-specific fusion network for MR Images

Multi-modal medical image fusion aims to integrate distinct imaging modalities to yield more comprehensive and precise medical images, which can benefit the subsequent image analysis tasks. However, prevailing state-of-the-art image fusion methods, despite substantial advancements, do not explicitly address the efficient handling of complementary information between modalities. Moreover, most current multi-modal medical image fusion methods encounter challenges when integrating with practical tasks, which lack the guidance of semantic information, thus hindering the generation of high-quality images for accurate identification of lesion areas. To deal with these challenges, this paper introduces a novel semantic information-guided modality-specific fusion network for multi-modal magnetic resonance (MR) images, named SIMFusion. Specifically, we propose the decomposition branch to capture common and specific features from MR images of different modalities for fusion to reduce information redundancy through correlated mutual information loss. Subsequently, we obtain their semantic characteristics via the semantic branch utilizing a pre-train segmentation network and, finally, achieve an adaptive balance between the two sets of features through a specialized fusion strategy. Extensive experiments demonstrate the superiority of SIMFusion over existing competing techniques on both BraTS2019 and ISLES2022 datasets, with 8.1% improvement in MI and 38.7% in VIFF in T2-T1ce image pairs, highlighting the potential of the proposed method as a promising solution for MR image fusion in practical applications. Our code will be released at https://github.com/Zhangxw-ustc/SIMFusion.

Language-vision matching for text-to-image synthesis with context-aware GAN

Text-to-image generation (T2I) aims to produce visually compelling images while maintaining a high degree of semantic consistency with textual descriptions. Despite the impressive progress made by existing methods, there are problems with limited details of synthesized images and insufficient correlation between the provided text description and the generated images. To address these issues, we propose a Context-Aware Generative Adversarial Network (CA-GAN), which generates images aligned with the input text representations. Specifically, the Context-Aware Block (CA-Block) learns a semantic-adaptive transformation based on text style, enabling the effective fusion of text descriptions and image features for high-quality image generation with better language-vision matching. Furthermore, we propose an Attention Convolution Module (ACM) to identify greater representative traits and avoid the inability to capitalize on non-local contextual information, which enables our model to generate images that exhibit numerous detailed attributes while maintaining high-quality and semantic consistency. Thereafter, we integrate self-attention with convolution to enhance feature maps to reinforce the semantic information in the discriminator, emphasizing critical feature channels while suppressing extraneous information, ultimately yielding more detailed and richer images. The experimental results demonstrate the superiority of our method over the SOTA approaches. In addition, further studies confirm the effectiveness of the generated visual details, which exhibit a high degree of alignment with the input text descriptions. Notably, our attention mechanism showcases cooperative effects contributing to overall performance improvement. The code is available at: https://github.com/hylneu/CAGAN.

AMPLIFYING AQUATIC VISUALS

Our proposed fusion-based approach aims to combat the degradation of underwater images, addressing issues such as diminished colors and indistinguishable objects. Leveraging contrast stretching and Auto White Balance, the technique significantly improves contrast and color, offering a straightforward yet effective solution to enhance visibility in aquatic images, crucial for applications like video surveillance in outdoor computer vision systems. This straightforward approach plays a pivotal role in enhancing image visibility, contributing significantly to applications like video surveillance and other outdoor computer vision systems. Our dehazing process builds upon two essential statistical observations related to haze-free images and haze. By applying dark channel prior and guided filter to the decomposed image, we effectively estimate atmosphere light, resulting in a dehazed output. This method tackles the intricacies of haze, contributing to the generation of high-quality images with improved details and clarity. Beyond image enhancement, our approach showcases versatility by extending into object classification, demonstrating the broader impact and potential of our method in the realm of outdoor computer vision systems.

FusionDiff: A unified image fusion network based on diffusion probabilistic models

This paper introduces FusionDiff, a novel unified end-to-end image fusion network based on diffusion probabilistic models. The proposed method addresses the data dependency issue present in current unified image fusion approaches. Initially, the method pre-fuses the source images after diffusion. Subsequently, a noise prediction network forecasts the noise applied to the pre-fused image. During this stage, the Spatially-Adaptive Constraint layer is employed to restrict the fusion image, thereby maximizing the preservation of respective source image features. Finally, in the reverse process, skip-sampling is employed to effectively overcome the computational time drawback of traditional diffusion probabilistic models while maintaining high-quality image generation. Compared to other diffusion model-based image fusion methods, our approach stands out for its structural simplicity and ease of training, serving as a unified image fusion method. Furthermore, compared to other unified image fusion methods, our proposed network fully leverages the generalization of the diffusion probabilistic model, achieving adaptive feature extraction by constraining the inverse process of source images. Qualitative and quantitative experimental results across four classic image fusion tasks demonstrate the superiority of our method over state-of-the-art unified image fusion methods in recent years.

Latent Diffusion Models for Structural Component Design

Recent advances in generative modeling, namely Diffusion models, have revolutionized generative modeling, enabling high-quality image generation tailored to user needs. This paper proposes a framework for the generative design of structural components. Specifically, we employ a Latent Diffusion model to generate potential designs of a component that can satisfy a set of problem-specific loading conditions. One of the distinct advantages our approach offers over other generative approaches is the editing of existing designs. We train our model using a dataset of geometries obtained from structural topology optimization utilizing the SIMP algorithm. Consequently, our framework generates inherently near-optimal designs. Our work presents quantitative results that support the structural performance of the generated designs and the variability in potential candidate designs. Furthermore, we provide evidence of the scalability of our framework by operating over voxel domains with resolutions varying from 323 to 1283. Our framework can be used as a starting point for generating novel near-optimal designs similar to topology-optimized designs.

Advancing generalizations of multi-scale GAN via adversarial perturbation augmentations

The multi-scale generative adversarial networks (MS-GANs) typically introduce a series of GANs to process hierarchical image patches from a single image, which suites image manipulation scenarios where it takes one image as input to produce manipulated results. However, during MS-GAN training, leveraging only a single image in hierarchical patches brings data insufficiency for model generalization, which limits MS-GAN to produce fine-grained texture details. To address this problem, we theoretically formulate MS-GAN generalization from the PAC-Bayes bound perspective in this work. Under this formulation, we obtain a non-vacuous upper bound of the generalization error, and propose an adversarial perturbation augmentation method named AP-Aug. Given an input image, our AP-Aug produces adversarial perturbations to augment the original image. The augmented training images benefit MS-GAN to approach the generalization upper bound. To this end, we advance the generalizations of MS-GAN to improve image generation performance. In the experiments, our AP-Aug benefits MS-GAN to achieve high quality image generation under several image manipulation scenarios including paint2image, image style transfer, and image super-resolution. The improvement shows that our AP-Aug advances the MS-GAN generalizations for high quality image generations, which perform favorably against state-of-the-art approaches.

Diffusion Probabilistic Modeling for Video Generation.

Denoising diffusion probabilistic models are a promising new class of generative models that mark a milestone in high-quality image generation. This paper showcases their ability to sequentially generate video, surpassing prior methods in perceptual and probabilistic forecasting metrics. We propose an autoregressive, end-to-end optimized video diffusion model inspired by recent advances in neural video compression. The model successively generates future frames by correcting a deterministic next-frame prediction using a stochastic residual generated by an inverse diffusion process. We compare this approach against six baselines on four datasets involving natural and simulation-based videos. We find significant improvements in terms of perceptual quality and probabilistic frame forecasting ability for all datasets.

Automatic Face Expression Neutralisation and Posture Correction via 3D Contrastive Learning

The performance of recognition systems can be significantly affected by various factors, with facial expression poses and lighting changes being the main confounding factors. In order to minimize their impact, we propose an intriguing method in this study that enables the generation of high-quality images specifically tailored to the target domain. Our objective is to utilize disentangled representation to effectively model the decomposition of data variations and generate neutral facial expression images with frontal posture and adaptive illumination. To achieve this, we incorporate 3D priors in the adversarial learning during the training process, simulating the generation of an analytical 3D face deformation as well as rendering operations. Additionally, we employ contrastive learning to control the disentanglement of the generated faces while preserving the essential properties of facial features. This technique enables us to learn an embedding space in which similar data samples are represented closely, while distinct samples are kept far apart from each other. Furthermore, we conduct an analysis of the learned latent space and introduce several other significant properties that enhance the reinforcement of factor disentanglement. These properties include an imitation learning algorithm, which facilitates the acquisition of meaningful patterns and characteristics.

Sketch2Photo: Synthesizing photo-realistic images from sketches via global contexts

Sketch-to-image synthesis aims to generate realistic images that match the input sketches or edge maps exactly. Most known sketch-to-image synthesis methods use various generative adversarial networks (GANs) that are trained with numerous pairs of sketches and real images. Because of the convolution locality, the low-level layers of the generators in these GANs lack global perception ability, causing feature maps derived from them easily to overlook global cues. Since the global receptive field is crucial for acquiring the non-local structures and features of sketches, the absence of global contexts will impact the generation of high-quality images. Some recent models turn to self-attention to construct global dependencies. However, they are not viable for large feature maps for the quadratic computational complexity concerning the size of feature maps. To address these problems, in this work, we propose Sketch2Photo — a new image synthesis approach that can capture global contexts as well as local features to generate photo-realistic images from weak or partial sketches or edge maps. We employ fast Fourier convolution (FFC) residual blocks to create global receptive fields in the bottom layers of the network and incorporate Swin Transformer block (STB) units to obtain long-range global contexts for large-size feature maps efficiently. We also present an improved spatial attention pooling (ISAP) module to relax the strict alignment requirements between incomplete sketches and generated images. Quantitative and qualitative experiments on multiple public datasets demonstrate the superiority of the proposed approach over many other sketch-to-image synthesis methods. The project code is available at https://github.com/hengliusky/Skecth2Photo.

Unsupervised Learning of Depth Estimation and Camera Pose With Multi-Scale GANs

Unsupervised learning methods have achieved remarkable performance in monocular depth estimation and camera pose, which mostly solve the multi-task learning problem by using their inner geometry consistency as the self-supervision signal. While most existing approaches mostly adopt the generative model to obtain the depth map prediction, so in the resolution of depth map there is room for improvement. To this end, we present our unsupervised learning architecture based on adversarial learning model, which is used for unsupervised learning of high-resolution single view depth and camera pose. Specifically, we present a multi-scale deep convolutional Generative Adversarial Network (GAN) based learning system, which consists of three networks (pose estimation network PCNN, Generator-D and Discriminator-D for depth map prediction). Furthermore, in order to generate high-resolution depth map, we propose a multi-scale GAN model (MSGAN) to decompose the hard high-quality image generation problem into more manageable sub-problems through a coarse-to-fine process. Then, we modify the overall generation architecture of GAN model by changing the down-sampling and up-sampling components to improve the quality and accuracy of the depth map prediction. Finally, in order to improve the rate of convergence, we use the Least Square Error to increase the penalty for outliers. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI dataset show that the proposed method provides better results for both pose estimation and depth recovery.

DeepAIA: An Automatic Image Annotation Model Based on Generative Adversarial Networks and Transfer Learning

Automatic image annotation (AIA) has been adopted in different applications such as image retrieval and classification. Deep Learning is used in AIA to extract image features and then convert these features into text descriptions and labels. However, conventional AIA models that employ deep learning methods suffer from various shortcomings, such as poor annotation performance. This work proposes an AIA model based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and transfer learning. GANs have attracted a lot of interest because of its ability to generate data without explicitly using probability density. Thus, it has proven its usefulness in image annotation and image augmentation. In this work, an Auxiliary classifier-GAN (ACGAN) has been used, where the discriminator predicts the class of an image rather than taking it as a given input; therefore, the stabilization of the training stage is ensured, and the generation of high-quality images is provided. Transfer learning is also used to enhance the performance of the classification. The proposed model outperforms the best state-of-the-art models in terms of MiAP, F-measure and error rate using ImageClef, ESPGame and IAPR-TC12 datasets.

X-ray tomography: the way from layer-by-layer radiography to computed tomography

The methods of X-ray computed tomography allow us to study the internal morphological structure of objects in a non-destructive way. The evolution of these methods is similar in many respects to the evolution of photography, where complex optics were replaced by mobile phone cameras, and the computers built into the phone took over the functions of high-quality image generation. X-ray tomography originated as a method of hardware non-invasive imaging of a certain internal cross-section of the human body. Today, thanks to the advanced reconstruction algorithms, a method makes it possible to reconstruct a digital 3D image of an object with a submicron resolution. In this article, we will analyze the tasks that the software part of the tomographic complex has to solve in addition to managing the process of data collection. The issues that are still considered open are also discussed. The relationship between the spatial resolution of the method, sensitivity and the radiation load is reviewed. An innovative approach to the organization of tomographic imaging, called “reconstruction with monitoring”, is described. This approach makes it possible to reduce the radiation load on the object by at least 2–3 times. In this work, we show that when X-ray computed tomography moves towards increasing the spatial resolution and reducing the radiation load, the software part of the method becomes increasingly important.

DTexFusion: Dynamic Texture Fusion Using a Consumer RGBD Sensor.

In addition to 3D geometry, accurate representation of texture is important when digitizing real objects in virtual worlds. Based on a single consumer RGBD sensor, accurate texture representation for static objects can be realized by fusing multi-frame information; however, extending the process to dynamic objects, which typically have time-varying textures, is difficult. Thus, to address this problem, we propose a compact keyframe-based representation that decouples a dynamic texture into a basic static texture and a set of multiplicative changing maps. With this representation, the proposed method first aligns textures recorded from multiple keyframes with the reconstructed dynamic geometry of the object. Errors in the alignment and geometry are then compensated in an innovative iterative linear optimization framework. With the reconstructed texture, we then employ a scheme to synthesize the dynamic object from arbitrary viewpoints. By considering temporal and local pose similarities jointly, dynamic textures in all keyframes are fused to guarantee high-quality image generation. Experimental results demonstrate that the proposed method handles various dynamic objects, including faces, bodies, cloth, and toys. In addition, qualitative and quantitative comparisons demonstrate that the proposed method outperforms state-of-the-art solutions.

Generative Adversarial Networks in Computer Vision

Generative adversarial networks (GANs) have been extensively studied in the past few years. Arguably their most significant impact has been in the area of computer vision where great advances have been made in challenges such as plausible image generation, image-to-image translation, facial attribute manipulation, and similar domains. Despite the significant successes achieved to date, applying GANs to real-world problems still poses significant challenges, three of which we focus on here. These are as follows: (1) the generation of high quality images, (2) diversity of image generation, and (3) stabilizing training. Focusing on the degree to which popular GAN technologies have made progress against these challenges, we provide a detailed review of the state-of-the-art in GAN-related research in the published scientific literature. We further structure this review through a convenient taxonomy we have adopted based on variations in GAN architectures and loss functions. While several reviews for GANs have been presented to date, none have considered the status of this field based on their progress toward addressing practical challenges relevant to computer vision. Accordingly, we review and critically discuss the most popular architecture-variant, and loss-variant GANs, for tackling these challenges. Our objective is to provide an overview as well as a critical analysis of the status of GAN research in terms of relevant progress toward critical computer vision application requirements. As we do this we also discuss the most compelling applications in computer vision in which GANs have demonstrated considerable success along with some suggestions for future research directions. Codes related to the GAN-variants studied in this work is summarized on https://github.com/sheqi/GAN_Review.

ML-CGAN: Conditional Generative Adversarial Network with a Meta-learner Structure for High-Quality Image Generation with Few Training Data

Since generative adversarial network (GAN) can learn data distribution and generate new samples based on the learned data distribution, it has become a research hotspot in the area of deep learning and cognitive computation. The learning of GAN heavily depends on a large set of training data. However, in many real-world applications, it is difficult to acquire a large number of data as needed. In this paper, we propose a novel generative adversarial network called ML-CGAN for generating authentic and diverse images with few training data. Particularly, ML-CGAN consists of two modules: the conditional generative adversarial network (CGAN) backbone and the meta-learner structure. The CGAN backbone is applied to generate images, while the meta-learner structure is an auxiliary network to provide deconvolutional weights for the generator of the CGAN backbone. Qualitative and quantitative experimental results on the MNIST, Fashion MNIST, CelebA and CIFAR-10 data sets demonstrate the superiority of ML-CGAN over state-of-the-art models. Specifically, the results show that the meta-learner structure can learn prior knowledge and transfer it to the new tasks, which is beneficial for generating authentic and diverse images in the new tasks with few training data.

Image generation from bounding box-represented semantic labels

Image generation with pixel-wise semantic information is suitable for the development of adversarial learning techniques. In this study, we propose a method for synthesizing objects with class-specific textures and fine-scale details based on bounding box-represented semantic labels. To achieve this goal, we note that the traditional generative adversarial network (GAN) uses noise as an input to generate realistic images with sufficient textures and details, but it cannot be guided by specific targets and requirements. By contrast, conditional GAN (cGAN) can involve various types of guiding information but it often ignores specific textures and details, thereby leading to less realistic results and low resolution. Thus, we propose a new translator-enhancer framework by combining cGAN and GAN to achieve high quality image generation. cGAN is used as a translator to match the semantic constraints whereas GAN is employed as an enhancer to provide details and textures. We also propose a new form of semantic label map as an input, which is represented by instance-level bounding boxes rather than segmentation masks. The semantic label map represented by bounding boxes makes it easier for users to provide the inputs and it also gives greater flexibility when generating object boundaries. The results obtained from qualitative and quantitative experiments showed that our method can generate realistic images of objects with semantic labels represented by bounding boxes. Our method can be used to generate images of novel scenes to support learning tasks during training with various scenes, which are difficult to capture in the real world.

Query is GAN: Scene Retrieval With Attentional Text-to-Image Generative Adversarial Network

Scene retrieval from input descriptions has been one of the most important applications with the increasing number of videos on the Web. However, this is still a challenging task since semantic gaps between features of texts and videos exist. In this paper, we try to solve this problem by utilizing a text-to-image Generative Adversarial Network (GAN), which has become one of the most attractive research topics in recent years. The text-to-image GAN is a deep learning model that can generate images from their corresponding descriptions. We propose a new retrieval framework, “Query is GAN”, based on the text-to-image GAN that drastically improves scene retrieval performance by simple procedures. Our novel idea makes use of images generated by the text-to-image GAN as queries for the scene retrieval task. In addition, unlike many studies on text-to-image GANs that mainly focused on the generation of high-quality images, we reveal that the generated images have reasonable visual features suitable for the queries even though they are not visually pleasant. We show the effectiveness of the proposed framework through experimental evaluation in which scene retrieval is performed from real video datasets.

Web-based medical data visualization and information sharing towards application in distributed diagnosis

Network based medical data computing and collaborative visualization have been commonly used in remote medicine and distributed diagnosis, where visualizing 3D medical data on web browsers and sharing medical information on internet are critically important. However, due to the lack of efficient algorithms and compatible graphics hardware support, there are still some major technical challenges in web based medical data visualization and information exploration on internet. To address these practical issues, we created a new network based medical data rendering and information sharing system, where an Apache HTTP Server was applied to handle data information, and MySQL and PHP were exploited for data storage and management. In this system, medical data rendering and computation were supported with GPU and WebGL 2.0 (WebGL2), and a novel data information extraction algorithm was designed to optimize data storage and management. Taking advantage of the new 3D features of WebGL2, a web based raycasting algorithm was developed to deliver real-time data visualization on web browsers, in which a novel voxel classification method was integrated for color mapping and high-quality image generation. The developed medical information system can deliver 60 ± 2 frames per second rendering performance for high-quality medical data visual exploration on modern browsers as well as medical information communication on internet. The system has been seamlessly integrated with web server, database and client computers equipped with modern graphics hardware, which has wide applications in the fields such as internet based computer-aided medical decision and education, as well as distributed disease diagnosis.