Image Generation Applications Research Articles

Tunable Privacy Risk Evaluation of Generative Adversarial Networks.

Generative machine learning models such as Generative Adversarial Networks (GANs) have been shown to be especially successful in generating realistic synthetic data in image and tabular domains. However, it has been shown that such generative models, as well as the generated synthetic data, can reveal information contained in their privacy-sensitive training data, and therefore must be carefully evaluated before being used. The gold standard method through which such privacy leakage can be estimated is simulating membership inference attacks (MIAs), in which an attacker attempts to learn whether a given sample was part of the training data of a generative model. The state-of-the art MIAs against generative models, however, rely on strong assumptions (knowledge of the exact training dataset size), or require a lot of computational power (to retrain many "surrogate" generative models), which make them hard to use in practice. In this work, we propose a technique for evaluating privacy risks in GANs which exploits the outputs of the discriminator part of the standard GAN architecture. We evaluate our attacks in terms of performance in two synthetic image generation applications in radiology and ophthalmology, showing that our technique provides a more complete picture of the threats by performing worst-case privacy risk estimation and by identifying attacks with higher precision than the prior work.

MedSegDiff-V2: Diffusion-Based Medical Image Segmentation with Transformer

The Diffusion Probabilistic Model (DPM) has recently gained popularity in the field of computer vision, thanks to its image generation applications, such as Imagen, Latent Diffusion Models, and Stable Diffusion, which have demonstrated impressive capabilities and sparked much discussion within the community. Recent investigations have further unveiled the utility of DPM in the domain of medical image analysis, as underscored by the commendable performance exhibited by the medical image segmentation model across various tasks. Although these models were originally underpinned by a UNet architecture, there exists a potential avenue for enhancing their performance through the integration of vision transformer mechanisms. However, we discovered that simply combining these two models resulted in subpar performance. To effectively integrate these two cutting-edge techniques for the Medical image segmentation, we propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities. Through comprehensive evaluation, our approach demonstrates superiority over prior state-of-the-art (SOTA) methodologies. Code is released at https://github.com/KidsWithTokens/MedSegDiff.

ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models

Personalizing generative models offers a way to guide image generation with user-provided references. Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. However, representing and editing specific visual attributes such as material, style, and layout remains a challenge, leading to a lack of disentanglement and editability. To address this problem, we propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low to high frequency information, providing a new perspective on representing, generating, and editing images. We develop the Prompt Spectrum Space P*, an expanded textual conditioning space, and a new image representation method called ProSpect. ProSpect represents an image as a collection of inverted textual token embeddings encoded from per-stage prompts, where each prompt corresponds to a specific generation stage (i.e., a group of consecutive steps) of the diffusion model. Experimental results demonstrate that P* and ProSpect offer better disentanglement and controllability compared to existing methods. We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout, achieving previously unattainable results from a single image input without fine-tuning the diffusion models. Our source code is available at https://github.com/zyxElsa/ProSpect.

Learning by competing: Competitive multi-generator based adversarial learning

Generative adversarial networks (GANs) have been extensively used for dozens of image enhancement and image translation applications, where several traditional and novel architectures have been and are still being introduced. However, the classical training protocols do not fully explore the significant potential of such architectures. In this paper, we propose a novel conditional GAN (cGAN) framework called competitive multi-generator based cGAN (CMcGAN) for more effective training progress and enhanced image generation. Different from classical adversarial learning protocols, the generators of a CMcGAN are not only challenged by the discriminator for improved performance, but also by other generators. This is achieved by a sharing mechanism, where the current performances of the competing generators are shared among each other, motivating each generator to perform better than the rest. Extensive experiments in several conditional image generation applications show the significant improvements achieved by several traditional architectures when trained following our proposed learning strategy. Moreover, these architectures, trained by our learning strategy, outperformed the state-of-the-art approaches proposed in each tested application.

WDIG: a wavelet domain image generation framework based on frequency domain optimization

In the end-to-end image generation task, the spatial domain of pixel space cannot explicitly separate the low-frequency general information such as texture and color from the high-frequency detail information such as structure and identity. The loss function calculated in the spatial domain fails to effectively constrain the maintenance of detail information, and the generated image quality is insufficient. In this paper, a wavelet domain image generation (WDIG) framework is proposed to preserve the frequency information of images, in which the loss functions are constructed in the pixel space and wavelet space. In the pixel space, the low-frequency and high-frequency characteristic information of the signal are obtained by setting the appropriate Gaussian kernel and adopting the Gaussian fuzzy method. The loss function of ell_{1} norm spatial domain is constructed for the low-frequency and high-frequency characteristic information. In the wavelet space, the corresponding channel sub-band coefficients are obtained by wavelet transform, and the image is explicitly separated into high-frequency information and low-frequency information. The ell_{1} norm frequency domain loss function is constructed respectively for the sub-band coefficients. The WDIG can constrain model training more accurately and optimize model more precisely, so as to better maintain the details and quality of generated image. The WDIG framework is evaluated in the image generation applications including style transfer, image translation and Generative Adversarial Nets (GAN) Inversion. Experimental results show that the WDIG framework can effectively retain the details of images and generate more realistic images, and improve the image quality of the above applications in image generation.

A descriptive framework for the field of deep learning applications in medical images

Deep learning in medical image analysis is a typical interdisciplinary application, which needs support and cooperation of computer techniques and medical experience, and has broad application prospects. Since 2017, the number of related published articles has increased exponentially, imposing a burden on the literature review in this field. In this survey, we clustered 2068 retrieved articles into 15 topics through Latent Dirichlet Allocation (LDA) and provided a rough overview on the application of deep learning in medical images. On this basis, we conducted a detailed review with 77 top representative articles. We built a descriptive review framework based on LDA and discussed classification, object detection, segmentation, and image generation applications in medical images for the field of deep learning from the perspective of image modalities. We ended with discussing current challenges and future research directions of deep learning in medical images analysis.

Two-dimensional optical scanning for large-field-of-view optical image generation applications

Optical image generation is a very important step in many optical imaging system applications. However, current optical image generation systems (OIGS) produce a fixed simple grid image with a small field of view (FOV) because of the physical limitation of the OIGS itself. Here, we use proposed an optical image generation method using optical scanners to achieve a large FOV and images defined by the user. The combination of two scanners implements the two-dimensional optical scanning for obtaining the improvement of field-of-views (FOV) in two-dimensional (2D) directions. Here, three important findings, including scanning optical configurations, an analog-to-digital-based scanning pattern, and a scanning imaging model, of 2D optical scanning in the image generation with a large field of view are reported. The proposed method can improve the FOV with 20° in the 2D directions. These three results are very valuable references for the application of image generation. The proposed method has potential applications in multiple fields, such as camera calibration, computation imaging, and image compression.

GNA: Reconfigurable and Efficient Architecture for Generative Network Acceleration

Generative networks have become ubiquitous in image generation applications like image super-resolution, image to image translation, and text to image synthesis. They are usually composed of convolutional (CONV) layers, convolution-based residual blocks, and deconvolutional (DeCONV) layers. Previous works on neural network acceleration focus too much on optimizing CONV layers computation such as data-reuse or parallel computation, but have low processing element (PE) utilization in computing residual blocks and DeCONV layers: residual blocks require very high memory bandwidth when performing elementwise additions on residual paths; DeCONV layers have imbalanced operation counts for different outputs. In this paper, we propose a dual convolution mapping method for CONV and DeCONV layers to make full use of the available PE resources. A cross-layer scheduling method is also proposed to avoid extra off-chip memory access in residual block processing. Precision-adaptive PEs and buffer bandwidth reconfiguration are used to support flexible bitwidths for both inputs and weights in deep neural networks. We implement a generative network accelerator (GNA) based on intra-PE processing, inter-PE processing, and cross-layer scheduling techniques. Owing to the proposed optimization techniques, GNA achieves energy efficiency of 2.05 TOPS/W with 61% higher PE utilization than traditional methods in generative network acceleration.