Conditional Generative Network Research Articles

Zero-Shot SAR Target Recognition Based on a Conditional Generative Network with Category Features from Simulated Images

SAR image target recognition relies heavily on a large number of annotated samples, making it difficult to classify the unseen class targets. Due to the lack of effective category auxiliary information, the current zero-shot target recognition methods for SAR images are limited to inferring only one unseen class rather than classifying multiple unseen classes. To address this issue, a conditional generative network with the category features from the simulated images for zero-shot SAR target recognition is proposed in this paper. Firstly, the deep features are extracted from the simulated images and fused into the category features that characterize the entire class. Then, a conditional VAE-GAN network is constructed to generate the feature instances of the unseen classes. The high-level semantic information shared in the category features aids in generalizing the mapping learned from the seen classes to the unseen classes. Finally, the generated features of the unseen classes are used to train a classifier that can classify the real unseen images. The classification accuracies for the targets of the three unseen classes based on the proposed method can reach 99.80 ± 1.22% and 71.57 ± 2.28% with the SAMPLE dataset and the MSTAR dataset, respectively. The advantage and validity of the proposed architecture are indicated with a small number of the seen classes and a small amount of the training data. Furthermore, the proposed method can be extended to generalized zero-shot recognition.

Data augmentation for invasive brain–computer interfaces based on stereo-electroencephalography (SEEG)

Objective. Deep learning is increasingly used for brain–computer interfaces (BCIs). However, the quantity of available data is sparse, especially for invasive BCIs. Data augmentation (DA) methods, such as generative models, can help to address this sparseness. However, all the existing studies on brain signals were based on convolutional neural networks and ignored the temporal dependence. This paper attempted to enhance generative models by capturing the temporal relationship from a time-series perspective. Approach. A conditional generative network (conditional transformer-based generative adversarial network (cTGAN)) based on the transformer model was proposed. The proposed method was tested using a stereo-electroencephalography (SEEG) dataset which was recorded from eight epileptic patients performing five different movements. Three other commonly used DA methods were also implemented: noise injection (NI), variational autoencoder (VAE), and conditional Wasserstein generative adversarial network with gradient penalty (cWGANGP). Using the proposed method, the artificial SEEG data was generated, and several metrics were used to compare the data quality, including visual inspection, cosine similarity (CS), Jensen–Shannon distance (JSD), and the effect on the performance of a deep learning-based classifier. Main results. Both the proposed cTGAN and the cWGANGP methods were able to generate realistic data, while NI and VAE outputted inferior samples when visualized as raw sequences and in a lower dimensional space. The cTGAN generated the best samples in terms of CS and JSD and outperformed cWGANGP significantly in enhancing the performance of a deep learning-based classifier (each of them yielding a significant improvement of 6% and 3.4%, respectively). Significance. This is the first time that DA methods have been applied to invasive BCIs based on SEEG. In addition, this study demonstrated the advantages of the model that preserves the temporal dependence from a time-series perspective.

TP-GAN: Simple Adversarial Network With Additional Player for Dense Depth Image Estimation

We present a simple yet robust monocular depth estimation technique by synthesizing a depth map image from a single RGB input image using the advantage of generative adversarial networks (GAN). We employ an additional sub-model termed refiner to extract local depth features, then combine it with the global scene information from the generator to improve the GAN’s performance compared to the standard GAN architectural scheme. Notably, the generator is the first player to learn to synthesize depth images. The second player, the discriminator, classifies the generated depth. In the meantime, the third player, the refiner, enhances the final reconstructed depth. Complementing the GAN model, we apply a conditional generative network (cGAN) to lead the generator in mapping the input image to the respective depth representation. We further incorporate a structured similarity (SSIM) as our loss function for the generator and refiner in GAN training. Through extensive experiment validation, we confirmed the performance of our strategy on the publicly indoor NYU Depth v2 and KITTI outdoor data. Experiment results on the NYU depth v2 dataset show that our proposed approach achieves the best performance by 96.0% on threshold accuracy ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\delta &lt; 1.25^{2}$ </tex-math></inline-formula> ) and the second-best accuracy on all thresholds on the KITTI dataset. We discovered that our proposed method compares favorably to numerous existing monocular depth estimation strategies and demonstrates a considerable improvement in the accuracy of image depth estimation despite its simple network architecture.

A novel data-driven optimal chiller loading regulator based on backward modeling approach

This paper proposes a new method termed backward modeling approach (BMA) to achieve optimal chiller loading (OCL) for reducing energy consumption in industries running multiple-chillers with different efficiency. The developed OCL regulator (OCLR) based on novel BMA approach is composed of conditional generative network (cGAN) and deep neural network (DNN). Most works on the optimal chiller loading problem are to find out the setting of partial load rate (PLR) for each chiller. However, PLR for each chiller cannot be controlled directly and can only be achieved through setting chilled water supply temperature. A novel feedback control framework was developed to identify the relationship between chilled water supply temperature and the PLR. In light of this, the control instruction for chilled water supply temperature can be set to achieve the desired energy saving. The practical feasibility of loading optimization based on developed OCLR was evaluated by conducting field validation for 1 year in a reputed panel manufacturing factory based in Taiwan running multiple-chiller system. From the experimental results, it is evident that the developed data-driven OCLR based on BMA has very high performance and was able to conserve significant energy in the range of 81.9 MWh to 198 MWh per year.

On-demand design based on deep learning and phase manipulation of all-silicon terahertz chiral metasurfaces

Chiral metasurfaces have been widely used in sensing, imaging and other fields because they can manipulate light through the efficient circular dichroism (CD). However, its on-demand design is still a very challenging task. In this work, we propose an on-demand multiple reverse design based on deep learning, named target-driven conditional generative network (TCGN). It can reverse design the metasurface structure that meets the required CD, and its mean square error (MAE) is 0.0089. We use this method to inversely design multiple sets of metasurfaces with different structures, and all their CD values can exceed 0.36. Both simulations and experiments prove the feasibility and effectiveness of using deep learning to reverse design metasurfaces. In addition, the designed metasurface can realize chiral wavefront control under dual frequency. This design method based on deep learning can rapidly and efficiently design the chiral metasurfaces, which provides a new way for the design of metasurfaces.

Image Rain Removal Using Conditional Generative Networks Incorporating

The research of removing rain from pictures or videos has always been an important topic in the field of computer vision and image processing. Most noise reduction methods more or less remove texture details in rain-free areas, resulting in an over-smoothing effect in the restored background. The research on image noise removal is very meaningful. We exploit the powerful generative power of a modified generative adversarial network (CGAN) by enforcing an additional condition that makes the derained image indistinguishable from its corresponding ground-truth clean image. An efficient and lightweight attention machine mechanism NAM is introduced in the generator, and an IDN-CGAN model is proposed to capture image salient features through attention operations. Taking advantage of the mutual information in different dimensions of the features to further suppress insignificant channels or pixels to ensure better visual quality, we also introduce a new fine-grained loss function in the generator-discriminator pair, predicting and real data degree of disparity to achieve improved results.

Probabilistic 4D predictive model from in-room surrogates using conditional generative networks for image-guided radiotherapy.

Shape and location organ variability induced by respiration constitutes one of the main challenges during dose delivery in radiotherapy. Providing up-to-date volumetric information during treatment can improve tumor tracking, thereby increasing treatment efficiency and reducing damage to healthy tissue. We propose a novel probabilistic model to address the problem of volumetric estimation with scalable predictive horizon from image-based surrogates during radiotherapy treatments, thus enabling out-of-plane tracking of targets. This problem is formulated as a conditional learning task, where the predictive variables are the 2D surrogate images and a pre-operative static 3D volume. The model learns a distribution of realistic motion fields over a population dataset. Simultaneously, a seq-2-seq inspired temporal mechanism acts over the surrogate images yielding extrapolated-in-time representations. The phase-specific motion distributions are associated with the predicted temporal representations, allowing the recovery of dense organ deformation in multiple times. Due to its generative nature, this model enables uncertainty estimations by sampling the latent space multiple times. Furthermore, it can be readily personalized to a new subject via fine-tuning, and does not require inter-subject correspondences. The proposed model was evaluated on free-breathing 4D MRI and ultrasound datasets from 25 healthy volunteers, as well as on 11 cancer patients. A navigator-based data augmentation strategy was used during the slice reordering process to increase model robustness against inter-cycle variability. The patient data was used as a hold-out test set. Our approach yields volumetric prediction from image surrogates with a mean error of 1.67 ± 1.68mm and 2.17 ± 0.82mm in unseen cases of the patient MRI and US datasets, respectively. Moreover, model personalization yields a mean landmark error of 1.4 ± 1.1mm compared to ground truth annotations in the volunteer MRI dataset, with statistically significant improvements over state-of-the-art.

A Rapid, End‐to‐end, Generative Model for Gaseous Phenomena from Limited Views

AbstractDespite the rapid development and proliferation of computer graphics hardware devices for scene capture in the most recent decade, the high‐resolution 3D/4D acquisition of gaseous scenes (e.g., smokes) in real time remains technically challenging in graphics research nowadays. In this paper, we explore a hybrid approach to simultaneously taking advantage of both the model‐centric method and the data‐driven method. Specifically, this paper develops a novel conditional generative model to rapidly reconstruct the temporal density and velocity fields of gaseous phenomena based on the sequence of two projection views. With the data‐driven method, we can achieve the strong coupling of density update and the estimation of flow motion, as a result, we can greatly improve the reconstruction performance for smoke scenes. First, we employ a conditional generative network to generate the initial density field from input projection views and estimate the flow motion based on the adjacent frames. Second, we utilize the differentiable advection layer and design a velocity estimation network with the long‐term mechanism to help achieve the end‐to‐end training and more stable graphics effects. Third, we can re‐simulate the input scene with flexible coupling effects based on the estimated velocity field subject to artists' guidance or user interaction. Moreover, our generative model could accommodate single projection view as input. In practice, more input projection views are enabling and facilitating the high‐fidelity reconstruction with more realistic and finer details. We have conducted extensive experiments to confirm the effectiveness, efficiency, and robustness of our new method compared with the previous state‐of‐the‐art techniques.

CloTH-VTON+: Clothing Three-Dimensional Reconstruction for Hybrid Image-Based Virtual Try-ON

Image-based virtual try-on (VTON) systems based on deep learning have attracted research and commercial interests. Although they show their strengths in blending the person and try-on clothing image and synthesizing the dis-occluded regions, their results for complex-posed persons are often unsatisfactory due to the limitations in their geometry deformation and texture-preserving capacity. To address these challenges, we propose CloTH-VTON+ for seamlessly integrating the image-based deep learning methods and the strength of the 3D model in shape deformation. Specifically, a fully automatic pipeline is developed for 3D clothing model reconstruction and deformation using a reference human model: first, the try-on clothing is matched to the target clothing regions in the simple shaped reference human model, and then the 3D clothing model is reconstructed. The reconstructed 3D clothing model can generate a very natural pose and shape transfer, retaining the textures of clothes. A clothing refinement network further refines the alignment, eliminating the misalignment due to the errors in human pose estimation and 3D deformation. The deformed clothing images are combined utilizing conditional generative networks to in-paint the dis-occluded areas and blend them all. Experiments on an existing benchmark dataset demonstrate that CloTH-VTON+ generates higher quality results in comparison to the state-of-the-art VTON systems and CloTH-VTON. CloTH-VTON+ can be incorporated into extended applications such as multi-pose guided and Video VTON.

Multiple Pose Virtual Try-On Based on 3D Clothing Reconstruction

Recently, image-based virtual try-on (VTON) systems using deep generative models have drawn significant research attention. However, the 2D clothing shape transform methods in the earlier works show serious limitations in 3D clothing deformation required in multiple-pose VTON scenarios. In this paper, we develop a 3D-MPVTON system of two pipelines and show that a 3D clothing model reconstruction approach provides much better results for the multi-pose VTON scenario. First, the 3D clothing model reconstruction pipeline is based on CloTH-VTON+. The try-on clothing is matched to the target clothing regions in the simply-shaped reference human model and its 3D model is reconstructed using the associated 3D human body model. For natural clothing rendering from an arbitrary view, accurate texture mapping is crucial. We have developed a highly accurate texture matching method. The try-on pipeline first generates target segmentation from the target pose for conditional information for the following stage’s network models. Correct target segmentation was one of the main performance bottlenecks in previous VTON studies. Our proposed equalized entropy loss for the target segmentation generation network greatly reduces the segmentation label imbalance, and results in high-quality segmentation and reduced training time. The rigged reconstructed 3D clothing model can be easily deformed into the target pose and the human body shape while retaining the textures of the clothes. The remaining parts, i.e., the non-target clothing regions of the human in the target pose are generated through a deep generative human pose transfer model. Finally, the generated clothing and the remaining parts are combined using conditional generative networks to in-paint the dis-occluded areas and blend them together. Our proposed hybrid pipeline outperforms the previous 2D-based approaches by substantial margins in both objective evaluation and subjective user study, especially in the cases of the large pose and viewpoint changes.

SAR Image Ship Object Generation and Classification With Improved Residual Conditional Generative Adversarial Network

Image generation by the conditional generative adversarial network (CGAN) can provide rich samples for supervised classification tasks. However, the unstable gradient updating problem makes it difficult to be used for image generation, especially for the high-resolution SAR image. In this letter, an improved residual condition generation network is proposed to enhance the quality of generated images and classification accuracy for hard negative examples. First, a residual convolutional-based block is built to clear the detail texture of different types of targets. Then, the gradient penalty and Wasserstein loss are used as discriminators to improve the similarity between real samples and the intra diversity of a generated images. We test the proposed method on generation and classification of three kinds of commercial ships using C-band 3-m Gaofen-3 (GF-3) SAR images. Experimental results show that our method can generate high-quality ship samples and achieve good classification accuracy.

Towards Photo-Realistic Facial Expression Manipulation

We present a method for photo-realistic face manipulation. Given a single RGB face image with an arbitrary expression, our method can synthesize another arbitrary expression of the same person. To achieve this, we first fit a 3D face model and disentangle the face into its texture and shape. We then train separate networks in each of these spaces. In texture space, we use a conditional generative network to change the appearance, and carefully design the input format and loss functions to achieve the best results. In shape space, we use a fully connected network to predict an accurate face shape. When available, the shape branch uses depth data for supervision. Both networks are conditioned on expression coefficients rather than discrete labels, allowing us to generate an unlimited number of expressions. Furthermore, we adopt spatially adaptive denormalization on our texture space representation to improve the quality of the synthesized results. We show the superiority of this disentangling approach through both quantitative and qualitative studies. The proposed method does not require paired data, and is trained using an in-the-wild dataset of videos consisting of talking people. To achieve this, we present a simple yet efficient method to select appropriate key frames from these videos. In a user study, our method is preferred in 83.2% of cases when compared to state-of-the-art alternative approaches.

Enabling Dark Energy Science with Deep Generative Models of Galaxy Images

Understanding the nature of dark energy, the mysterious force driving the accelerated expansion of the Universe, is a major challenge of modern cosmology. The next generation of cosmological surveys, specifically designed to address this issue, rely on accurate measurements of the apparent shapes of distant galaxies. However, shape measurement methods suffer from various unavoidable biases and therefore will rely on a precise calibration to meet the accuracy requirements of the science analysis. This calibration process remains an open challenge as it requires large sets of high quality galaxy images. To this end, we study the application of deep conditional generative models in generating realistic galaxy images. In particular we consider variations on conditional variational autoencoder and introduce a new adversarial objective for training of conditional generative networks. Our results suggest a reliable alternative to the acquisition of expensive high quality observations for generating the calibration data needed by the next generation of cosmological surveys.