Problem Of Image Generation Research Articles

Image Composition Method Based on a Spatial Position Analysis Network

Realistic image composition aims to composite new images by fusing a source object into a target image. It is a challenging problem due to the complex multi-task framework, including sensible object placement, appearance consistency, shadow generation, etc. Most existing researchers attempt to address one of the issues. Especially before compositing, there is no matching assignment between the source object and target image, which often leads to unreasonable results. To address the issues above, we consider image composition as an image generation problem and propose a deep adversarial learning network via spatial position analysis. We target the analysis network segment and classify the objects in target images. A spatial alignment network matches the segmented objects with the source objects, and predicts a sensible placement position, and an adversarial network generates a realistic composite image with the shadow and reflection of the source object. Furthermore, we use the classification information of target objects to filter out unreasonable image compositing. Moreover, we introduce a new test set to evaluate the network generalization for our multi-task image composition dataset. Extensive experimental results of the SHU (Shanghai University) dataset demonstrate that our deep spatial position analysis network remarkably enhances the compositing performance in realistic, shadow, and reflection generations.

Training Generative Adversarial Networks via Stochastic Nash Games.

Generative adversarial networks (GANs) are a class of generative models with two antagonistic neural networks: a generator and a discriminator. These two neural networks compete against each other through an adversarial process that can be modeled as a stochastic Nash equilibrium problem. Since the associated training process is challenging, it is fundamental to design reliable algorithms to compute an equilibrium. In this article, we propose a stochastic relaxed forward-backward (SRFB) algorithm for GANs, and we show convergence to an exact solution when an increasing number of data is available. We also show convergence of an averaged variant of the SRFB algorithm to a neighborhood of the solution when only a few samples are available. In both cases, convergence is guaranteed when the pseudogradient mapping of the game is monotone. This assumption is among the weakest known in the literature. Moreover, we apply our algorithm to the image generation problem.

DRGAN: A dual resolution guided low-resolution image inpainting

Although image inpainting is a challenging task in computer vision, most existing image inpainting methods have achieved remarkable progress. However, occlusion and low resolution often appear on one image simultaneously in the real world, and only a few methods can address low-resolution image inpainting. To tackle this problem, we propose a novel method called the Dual Resolution Generative Adversarial Network (DRGAN), which formulates low-resolution (LR) image inpainting as a constrained image generation problem. The proposed DRGAN first trains a generative network to obtain high-quality images from noise vectors. Then a dual-resolution loss is designed to effectively integrate both high- and low-resolution information to optimize the input vectors initialized by random noise to generate high-quality images. Compared with the model that learns the mapping function from low- to high-quality images, the proposed model maps input noise vectors to images and can thus adapt more effectively to various types of degradation and large occlusions. Extensive experiments on synthetic LR data and real-world images demonstrate the effectiveness of the proposed DRGAN.

Automatic Creation of Quality Images from Text using Multiple Generative Adversial Network

Automatic creation of realistic images is a tedious process even though the state-of-the-art AI/ML algorithms are employed. There is a lot of demand for such automatic image generators that could create high quality images. Many have this problem of visualizing things from the explanations they hear about. Thus, the text to image generation problem is necessary because it has significant applications in CAD, art generation and many more. This is a challenging task since the image should be realistic and consistent with the text. One of the most common uses of modern conditional generative models is the generation of visuals from natural language. Viewing an image makes us easily understand what that image is, rather than hearing someone describing the image. To bridge the semantic gap between text and image, Generative Adversarial Network (GAN) systems are used to achieve high accuracy. The proposed system helps in generation of superior quality images that are meaningfully consistent with the text.

Deep Label Prior: Pre-Training-Free Salient Object Detection Network Based on Label Learning

Due to the excellent semantics extraction capabilities, deep learning methods have significantly progressed in salient object detection (SOD). However, these methods often require time-consuming pre-training and large training datasets with ground truth. To address these issues, by referring to the framework known as “deep image prior (DIP),” we propose a SOD method called deep label prior network (DLPNet), which consists of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\cal A$</tex-math></inline-formula> -stream and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\cal B$</tex-math></inline-formula> -stream. The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\cal A$</tex-math></inline-formula> -stream includes two cascaded UNets and a simple CNNs module to extract the initial saliency map, while the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\cal B$</tex-math></inline-formula> -stream contains only two cascaded UNets, which refines the extracted initial saliency map. Unlike most of the current deep learning methods, DLPNet views the SOD task as a conditional image generation problem, relying on only the internal prior of the input itself to generate the saliency map. Hence, our DLPNet does not require pre-training or large annotated / unannotated datasets. Furthermore, we propose a morphology operation scheme, which creates rich pseudo-labels for facilitating the updating of network weights. Extensive experiments demonstrate that our method outperforms state-of-the-art unsupervised techniques and is even comparable to state-of-the-art supervised and weakly supervised methods on different evaluation metrics.

Geometry-Guided Street-View Panorama Synthesis From Satellite Imagery.

This paper presents a new approach for synthesizing a novel street-view panorama given a satellite image, as if captured from the geographical location at the center of the satellite image. Existing works approach this as an image generation problem, adopting generative adversarial networks to implicitly learn the cross-view transformations, but ignore the geometric constraints. In this paper, we make the geometric correspondences between the satellite and street-view images explicit so as to facilitate the transfer of information between domains. Specifically, we observe that when a 3D point is visible in both views, and the height of the point relative to the camera is known, there is a deterministic mapping between the projected points in the images. Motivated by this, we develop a novel satellite to street-view projection (S2SP) module which learns the height map and projects the satellite image to the ground-level viewpoint, explicitly connecting corresponding pixels. With these projected satellite images as input, we next employ a generator to synthesize realistic street-view panoramas that are geometrically consistent with the satellite images. Our S2SP module is differentiable and the whole framework is trained in an end-to-end manner. Extensive experimental results on two cross-view benchmark datasets demonstrate that our method generates more accurate and consistent images than existing approaches.

Unsupervised Learning of Depth Estimation and Camera Pose With Multi-Scale GANs

Unsupervised learning methods have achieved remarkable performance in monocular depth estimation and camera pose, which mostly solve the multi-task learning problem by using their inner geometry consistency as the self-supervision signal. While most existing approaches mostly adopt the generative model to obtain the depth map prediction, so in the resolution of depth map there is room for improvement. To this end, we present our unsupervised learning architecture based on adversarial learning model, which is used for unsupervised learning of high-resolution single view depth and camera pose. Specifically, we present a multi-scale deep convolutional Generative Adversarial Network (GAN) based learning system, which consists of three networks (pose estimation network PCNN, Generator-D and Discriminator-D for depth map prediction). Furthermore, in order to generate high-resolution depth map, we propose a multi-scale GAN model (MSGAN) to decompose the hard high-quality image generation problem into more manageable sub-problems through a coarse-to-fine process. Then, we modify the overall generation architecture of GAN model by changing the down-sampling and up-sampling components to improve the quality and accuracy of the depth map prediction. Finally, in order to improve the rate of convergence, we use the Least Square Error to increase the penalty for outliers. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI dataset show that the proposed method provides better results for both pose estimation and depth recovery.

PetsGAN: Rethinking Priors for Single Image Generation

Single image generation (SIG), described as generating diverse samples that have the same visual content as the given natural image, is first introduced by SinGAN, which builds a pyramid of GANs to progressively learn the internal patch distribution of the single image. It shows excellent performance in a wide range of image manipulation tasks. However, SinGAN has some limitations. Firstly, due to lack of semantic information, SinGAN cannot handle the object images well as it does on the scene and texture images. Secondly, the independent progressive training scheme is time-consuming and easy to cause artifacts accumulation. To tackle these problems, in this paper, we dig into the single image generation problem and improve SinGAN by fully-utilization of internal and external priors. The main contributions of this paper include: 1) We interpret single image generation from the perspective of the general generative task, that is, to learn a diverse distribution from the Dirac distribution composed of a single image. In order to solve this non-trivial problem, we construct a regularized latent variable model to formulate SIG. To the best of our knowledge, it is the first time to give a clear formulation and optimization goal of SIG, and all the existing methods for SIG can be regarded as special cases of this model. 2) We design a novel Prior-based end-to-end training GAN (PetsGAN), which is infused with internal prior and external prior to overcome the problems of SinGAN. For one thing, we employ the pre-trained GAN model to inject external prior for image generation, which can alleviate the problem of lack of semantic information and generate natural, reasonable and diverse samples, even for the object image. For another, we fully-utilize the internal prior by a differential Patch Matching module and an effective reconstruction network to generate consistent and realistic texture. 3) We construct abundant of qualitative and quantitative experiments on three datasets. The experimental results show our method surpasses other methods on both generated image quality, diversity, and training speed. Moreover, we apply our method to other image manipulation tasks (e.g., style transfer, harmonization) and the results further prove the effectiveness and efficiency of our method.

Exploring DeshuffleGANs in Self-Supervised Generative Adversarial Networks

Generative Adversarial Networks (GANs) have become the most used networks towards solving the problem of image generation. Self-supervised GANs are later proposed to avoid the catastrophic forgetting of the discriminator and to improve the image generation quality without needing the class labels. However, the generalizability of the self-supervision tasks on different GAN architectures is not studied before. To that end, we extensively analyze the contribution of a previously proposed self-supervision task, deshuffling of the DeshuffleGANs in the generalizability context. We assign the deshuffling task to two different GAN discriminators and study the effects of the task on both architectures. We extend the evaluations compared to the previously proposed DeshuffleGANs on various datasets. We show that the DeshuffleGAN obtains the best FID results for several datasets compared to the other self-supervised GANs. Furthermore, we compare the deshuffling with the rotation prediction that is firstly deployed to the GAN training and demonstrate that its contribution exceeds the rotation prediction. We design the conditional DeshuffleGAN called cDeshuffleGAN to evaluate the quality of the learnt representations. Lastly, we show the contribution of the self-supervision tasks to the GAN training on the loss landscape and present that the effects of these tasks may not be cooperative to the adversarial training in some settings. Our code can be found at https://github.com/gulcinbaykal/DeshuffleGAN.

Spatial Coevolution for Generative Adversarial Network Training

Generative Adversarial Networks (GANs) are difficult to train because of pathologies such as mode and discriminator collapse. Similar pathologies have been studied and addressed in competitive evolutionary computation by increased diversity. We study a system, Lipizzaner, that combines spatial coevolution with gradient-based learning to improve the robustness and scalability of GAN training. We study different features of Lipizzaner’s evolutionary computation methodology. Our ablation experiments determine that communication, selection, parameter optimization, and ensemble optimization each, as well as in combination, play critical roles. Lipizzaner succumbs less frequently to critical collapses and, as a side benefit, demonstrates improved performance. In addition, we show a GAN-training feature of Lipizzaner: the ability to train simultaneously with different loss functions in the gradient descent parameter learning framework of each GAN at each cell. We use an image generation problem to show that different loss function combinations result in models with better accuracy and more diversity in comparison to other existing evolutionary GAN models. Finally, Lipizzaner with multiple loss function options promotes the best model diversity while requiring a large grid size for adequate accuracy.

Product innovation concept generation based on deep learning and Kansei engineering

Industrial designers often present their initial concepts as design sketches. Rapid creation of new product conceptual images that meet users’ affective preferences remains challenging in real design environments. However, few published works in affective design directly assist industrial designers in creating product conceptual images. Thus, we propose a product concept generation approach framework based on deep learning and Kansei engineering (PCGA-DLKE) to assist industrial designers. Our work focuses on dataset collection, pre-processing, affective preferences recognition, conceptual image generation model and product style transfer networks. To mark users’ affective preferences, we established an affective recognition model by Kansei engineering and deep convolutional neural networks. To address the product conceptual image generation problem, we proposed a product design GAN model (PD-GAN), generating product conceptual images with affective preferences. An improved fast neural style transfer network was successfully trained to meet users’ style preferences. This study aims to assist industrial designers in finding innovative concepts with affective preference. The Kansei evaluation shows that the innovation of the new product concept has been enhanced, indicating that the approach can better assist industrial designers in creating designs that meet users’ emotional needs. Hand drill design and bicycle helmet design are taken as a case study.

Cross-Modal Semantic Matching Generative Adversarial Networks for Text-to-Image Synthesis

Synthesizing photo-realistic images based on text descriptions is a challenging image generation problem. Although many recent approaches have significantly advanced the performance of text-to-image generation, to guarantee semantic matchings between the text description and synthesized image remains very challenging. In this paper, we propose a new model, Cross-modal Semantic Matching Generative Adversarial Networks (CSM-GAN), to improve the semantic consistency between text description and synthesized image for a fine-grained text-to-image generation. Two new modules are proposed in CSM-GAN: Text Encoder Module (TEM) and Textual-Visual Semantic Matching Module (TVSMM). TVSMM is aimed at making the distance of the pairs of synthesized image and its corresponding text description closer, in global semantic embedding space, than those of mismatched pairs. This improves the semantic consistency and consequently, the generalizability of CSM-GAN. In TEM, we introduce Text Convolutional Neural Networks (Text_CNNs) to capture and highlight local visual features in textual descriptions. Thorough experiments on two public benchmark datasets demonstrated the superiority of CSM-GAN over other representative state-of-the-art methods.

Physics Validation of Novel Convolutional 2D Architectures for Speeding Up High Energy Physics Simulations

The precise simulation of particle transport through detectors remains a key element for the successful interpretation of high energy physics results. However, Monte Carlo based simulation is extremely demanding in terms of computing resources. This challenge motivates investigations of faster, alternative approaches for replacing the standard Monte Carlo technique.We apply Generative Adversarial Networks (GANs), a deep learning technique, to replace the calorimeter detector simulations and speeding up the simulation time by orders of magnitude. We follow a previous approach which used three-dimensional convolutional neural networks and develop new two-dimensional convolutional networks to solve the same 3D image generation problem faster. Additionally, we increased the number of parameters and the neural networks representational power, obtaining a higher accuracy. We compare our best convolutional 2D neural network architecture and evaluate it versus the previous 3D architecture and Geant4 data. Our results demonstrate a high physics accuracy and further consolidate the use of GANs for fast detector simulations.

An Advanced Deep Generative Framework for Temporal Link Prediction in Dynamic Networks.

Temporal link prediction in dynamic networks has attracted increasing attention recently due to its valuable real-world applications. The primary challenge of temporal link prediction is to capture the spatial-temporal patterns and high nonlinearity of dynamic networks. Inspired by the success of image generation, we convert the dynamic network into a sequence of static images and formulate the temporal link prediction as a conditional image generation problem. We propose a novel deep generative framework, called NetworkGAN, to tackle the challenging temporal link prediction task efficiently, which simultaneously models the spatial and temporal features in the dynamic networks via deep learning techniques. The proposed NetworkGAN inherits the advantages of the graph convolutional network (GCN), the temporal matrix factorization (TMF), the long short-term memory network (LSTM), and the generative adversarial network (GAN). Specifically, an attentive GCN is first designed to automatically learn the spatial features of dynamic networks. Second, we propose a TMF enhanced attentive LSTM (TMF-LSTM) to capture the temporal dependencies and evolutionary patterns of dynamic networks, which predicts the network snapshot at next timestamp based on the network snapshots observed at previous timestamps. Furthermore, we employ a GAN framework to further refine the performance of temporal link prediction by using a discriminative model to guide the training of the deep generative model (i.e., TMF-LSTM) in an adversarial process. To verify the effectiveness of the proposed model, we conduct extensive experiments on five real-world datasets. Experimental results demonstrate the significant advantages of NetworkGAN compared to other strong competitors.

Deep Markov models of multidimensional random fields

Abstract The paper discusses the use of a doubly stochastic random field model based on autoregressions with multiple roots of characteristic equations for solving the problem of image generation and segmentation. It was proposed to form an image taking into account hidden Markov parameter fields. In particular, the correlation coefficients of the model, as well as the average values and brightness variances, are realizations of random fields, the assessment of which in explicit form is not possible. In addition, a modification of a variational autoencoder for a doubly stochastic model is considered. Finally, based on the use of a modified variational auto encoder, classification and segmentation of satellite images is performed. The combined use of the developed segmentation algorithm with the Unet neural network provides an increase in segmentation accuracy of 3%.

Deep Visual Sharing With Colorblind

Visual sharing between color vision deficiency (CVD) and normal-vision audiences is challenging due to the need of simultaneous satisfaction of multiple binocular visual requirements, in order to offer a color-distinguishable and binocularly fusible visual experience to CVD audiences, without hurting the visual experience of the normal-vision audiences. Existing methods enable the feasibility of visual sharing but are not quite suitable for practical usage due to their instable and time-consuming optimization nature. In this paper, we propose the first deep-learning based solution for solving this visual sharing problem. Our method outperforms the existing solution in terms of all evaluation metrics. To achieve this, we propose to formulate this binocular image generation problem as a generation problem of a difference image , which can effectively enforce the binocular constraints. We also propose to retain only high-quality training data and enrich the variety of training data via intentionally synthesizing various confusing color combinations. With these, we train up a high-quality neural network model. Through multiple quantitative measurements and user study, we demonstrate this learning-based approach can significantly improve the quality of generated results with fast performance.

Gibbs-ringing artifact suppression with knowledge transfer from natural images to MR images

Gibbs-ringing is a common artifact in magnetic resonance imaging (MRI), which is mainly caused by the finite k-space sampling and the truncation of high frequency (HF) information at the sampling border. It is especially visible for imaging acquisitions at low resolution and can be typically suppressed by filtering at the expense of further loss of HF components. As a classic image restoration problem in MRI, Gibbs-ringing artifact suppression can be viewed as a typical ill-posed inverse problem of image generation in computer vision community, such as image super-resolution and inpainting. Inspired by this, the present work presents a novel method to suppress the Gibbs-ringing artifacts with knowledge transfer from natural images to MR images. The highly nonlinear relation between the artifact-degraded image and the corresponding artifact-free counterpart is modeled with a typical convolutional neural network (CNN), which is first trained with natural images and then fine-tuned with MR images. Unlike many other works, we use transfer learning between different types of images to deal with regression problems, rather than classification problems. The experimental results exhibit that there exists information sharing between natural images and MR images with regard to the same problem, and the knowledge learned from natural images can indeed improve the performance of regression models on MR images.

Image and its manufacturing in human communication vis-à-vis the translation profession: the image dynamism of the Polish sworn translator of the English language

It is claimed that the Polish sworn translator in his/her translating activities is subject to a complex processes of image manufacturing. Based on some important contributions to the problem of image generation, it is shown that the Polish sworn translator’s practices within image generation suffer from being intuitive, not very careful and lacking cautious planning. It is also claimed that a successful image generation should be strengthened by proper cross-cultural conditioning.

Computational modeling of cellular structures using conditional deep generative networks.

Cellular function is closely related to the localizations of its sub-structures. It is, however, challenging to experimentally label all sub-cellular structures simultaneously in the same cell. This raises the need of building a computational model to learn the relationships among these sub-cellular structures and use reference structures to infer the localizations of other structures. We formulate such a task as a conditional image generation problem and propose to use conditional generative adversarial networks for tackling it. We employ an encoder-decoder network as the generator and propose to use skip connections between the encoder and decoder to provide spatial information to the decoder. To incorporate the conditional information in a variety of different ways, we develop three different types of skip connections, known as the self-gated connection, encoder-gated connection and label-gated connection. The proposed skip connections are built based on the conditional information using gating mechanisms. By learning a gating function, the network is able to control what information should be passed through the skip connections from the encoder to the decoder. Since the gate parameters are also learned automatically, we expect that only useful spatial information is transmitted to the decoder to help image generation. We perform both qualitative and quantitative evaluations to assess the effectiveness of our proposed approaches. Experimental results show that our cGAN-based approaches have the ability to generate the desired sub-cellular structures correctly. Our results also demonstrate that the proposed approaches outperform the existing approach based on adversarial auto-encoders, and the new skip connections lead to improved performance. In addition, the localizations of generated sub-cellular structures by our approaches are consistent with observations in biological experiments. The source code and more results are available at https://github.com/divelab/cgan/.

Algorithms for simple object reconstruction using the largest possible object approach

SUMMARYRecently, three-dimensional motion analysis and shape recovery have attracted growing attention as promising avenues of approach to image understanding, object reconstruction as well as computer vision for robotic Systems. The image generation problem and the model generation problem are presented. More specifically, the inputs to the image generator are an old image, object model, motion specification, and hidden line and hidden surface algorithms. The output is a new image. Since the object model is given, the top-down approach is usually used. On the other hand, for the model generation problem, the input is an image sequence while the output is an object model. Since the object model is not given, and bottom-up approach is usually used.In this paper, the largest possible object approach is proposed and the advantages of this approach are stated. They are:1. This approach may be applicable to objects with planar surfaces as well as nonplanar surfaces.2. This approach may be applicable to the case that there are more than one face change per frame.3. This approach may be applicable when the camera is moving.4. This approach may be applicable when the object is measured by several measuring stations.By using this approach, algorithms for simple object reconstruction given a sequence of pictures are presented together with illustrative examples. The relevance and importance of this work are discussed.The results of this paper may have useful applications in object reconstruction, pictorial data reduction and computer vision for robotic Systems.