GAN（Generative Adversarial Nets）

Inspired by the adversarial learning in generative adversarial network, a novel optimization framework named Generative Adversarial Optimization (GAO) is proposed in this paper. This GAO framework sets up generative models to generate candidate solutions via an adversarial process, in which two models are trained alternatively and simultaneously, i.e., a generative model for generating candidate solutions and a discriminative model for estimating the probability that a generated solution is better than a current solution. The training procedure of the generative model is to maximize the probability of the discriminative model. Specifically, the generative model and the discriminative model are in this paper implemented by multi-layer perceptrons that can be trained by the back-propagation approach. As of an implementation of the proposed GAO, for the purpose of increasing the diversity of generated solutions, a guiding vector ever introduced in guided fireworks algorithm (GFWA) has been employed here to help constructing generated solutions for the generative model. Experiments on CEC2013 benchmark suite show that the proposed GAO framework achieves better than the state-of-art performance on multi-modal functions.

The generative model and discriminative model are the two categories of statistical models used in keystroke biometric areas. Generative models have the trait of handling missing or irregular data, and perform well for limited training data. Discriminative models are fast in making predictions for new data, resulting in faster classification of new data compared to the generative models. In an attempt to build an efficient model for keystroke biometric user identification, this study proposes a hybrid POHMM/SVM method taking advantage of both generative and discriminative models. The partially observable hidden Markov model (POHMM) is an extension of the hidden Markov model (HMM), which has shown promising performance in user verification and handling missing or infrequent data. On the other hand, the support vector machine (SVM) has been a widely used discriminative model in keystroke biometric systems for the last decade and achieved a higher accuracy rate for large data sets. In the proposed model, features are extracted using the POHMM model, and a one-class support vector machine is used as the anomaly detector. For user identification, the study examines POHMM parameters using five different discriminative classifiers: support vector machines, k-nearest neighbor, random forest, multilayer perceptron (MLP) neural network, and logistic regression. The best accuracy of 91.3% (mean 0.868, SD 0.132) is achieved by the proposed hybrid POHMM/SVM approach among all generative and discriminative models.

An important challenge for neural networks is to learn incrementally, i.e., learn new classes without catastrophic forgetting. To overcome this problem, generative replay technique has been suggested, which can generate samples belonging to learned classes while learning new ones. However, such generative models usually suffer from increased distribution mismatch between the generated and original samples along the learning process. In this work, we propose DeepCollaboration (D-Collab), a collaborative framework of deep generative and discriminative models to solve this problem effectively. We develop a discriminative learning model to incrementally update the latent feature space for continual classification. At the same time, a generative model is introduced to achieve conditional generation using the latent feature distribution produced by the discriminative model. Importantly, the generative and discriminative models are connected through bidirectional training to enforce cycle-consistency of mappings between feature and image domains. Furthermore, a domain alignment module is used to eliminate the divergence between the feature distributions of generated images and real ones. This module together with the discriminative model can perform effective sample mining to facilitate incremental learning. Extensive experiments on several visual recognition datasets show that our system can achieve state-of-the-art performance.

Plausibility-promoting generative adversarial network for abstractive text summarization with multi-task constraint

Generative models can be used for a wide range of tasks, and have the appealing ability to learn from both labelled and unlabelled data. In contrast, discriminative models cannot learn from unlabelled data, but tend to outperform their generative counterparts in supervised tasks. We develop a framework to jointly train deep generative and discriminative models, enjoying the benefits of both. The framework allows models to learn from labelled and unlabelled data, as well as naturally account for uncertainty in predictive distributions, providing the first Bayesian approach to semi-supervised learning with deep generative models. We demonstrate that our blended discriminative and generative models outperform purely generative models in both predictive performance and uncertainty calibration in a number of semi-supervised learning tasks.

Since discriminative models are usually constructed by learning a set of given training data, it is impossible to guarantee the predictive performance of data that are not generated from the same distribution as the training data. Such data are known as out-of-distribution (OOD) data, while data that follow the same distribution as the training data are referred to as in-distribution data. For practical applications, it is important to detect the OOD data before they are input into highly qualified classifiers. Recently, a likelihood ratio-based method of OOD detection has been proposed. In this method, the likelihood ratio calculated using two generative models with different noise conditions functions as a detection index to evaluate semantic information only, ignoring the background information associated with all classes of data. Here, the generative model used for OOD detection should estimate the true distribution of in-distribution data accurately; all classes are estimated together, using the conventional method. However, in-distribution data may follow a simpler distribution if each class is estimated separately; these simple distribution structures are easier to learn. For this reason, estimation accuracy can be improved by models that estimate the distribution structure of each class independently. In this study, we propose an OOD detection method that uses generative models trained independently for each class. We also conduct evaluation experiments, using image datasets to demonstrate the effectiveness of the proposed method.

Recent artificial intelligence research has witnessed great interest in automatically generating text descriptions of images, which are known as the image captioning task. Remarkable success has been achieved on domains where a large number of paired data in multimedia are available. Nevertheless, annotating sufficient data is labor-intensive and time-consuming, establishing significant barriers for adapting the image captioning systems to new domains. In this study, we introduc a novel Multitask Learning Algorithm for cross-Domain Image Captioning (MLADIC). MLADIC is a multitask system that simultaneously optimizes two coupled objectives via a dual learning mechanism: image captioning and text-to-image synthesis, with the hope that by leveraging the correlation of the two dual tasks, we are able to enhance the image captioning performance in the target domain. Concretely, the image captioning task is trained with an encoder-decoder model (i.e., CNN-LSTM) to generate textual descriptions of the input images. The image synthesis task employs the conditional generative adversarial network (C-GAN) to synthesize plausible images based on text descriptions. In C-GAN, a generative model $G$ synthesizes plausible images given text descriptions, and a discriminative model $D$ tries to distinguish the images in training data from the generated images by $G$. The adversarial process can eventually guide $G$ to generate plausible and high-quality images. To bridge the gap between different domains, a two-step strategy is adopted in order to transfer knowledge from the source domains to the target domains. First, we pre-train the model to learn the alignment between the neural representations of images and that of text data with the sufficient labeled source domain data. Second, we fine-tune the learned model by leveraging the limited image-text pairs and unpaired data in the target domain. We conduct extensive experiments to evaluate the performance of MLADIC by using the MSCOCO as the source domain data, and using Flickr30k and Oxford-102 as the target domain data. The results demonstrate that MLADIC achieves substantially better performance than the strong competitors for the cross-domain image captioning task.

When labelled training data is plentiful, discriminative techniques are widely used since they give excellent generalization performance. However, for large-scale applications such as object recognition, hand labelling of data is expensive, and there is much interest in semi-supervised techniques based on generative models in which the majority of the training data is unlabelled. Although the generalization performance of generative models can often be improved by 'training them discriminatively', they can then no longer make use of unlabelled data. In an attempt to gain the benefit of both generative and discriminative approaches, heuristic procedure have been proposed [2, 3] which interpolate between these two extremes by taking a convex combination of the generative and discriminative objective functions. In this paper we adopt a new perspective which says that there is only one correct way to train a given model, and that a 'discriminatively trained' generative model is fundamentally a new model [7]. From this viewpoint, generative and discriminative models correspond to specific choices for the prior over parameters. As well as giving a principled interpretation of 'discriminative training', this approach opens door to very general ways of interpolating between generative and discriminative extremes through alternative choices of prior. We illustrate this framework using both synthetic data and a practical example in the domain of multi-class object recognition. Our results show that, when the supply of labelled training data is limited, the optimum performance corresponds to a balance between the purely generative and the purely discriminative.

An interesting question surrounding semi-supervised learning for NLP is: should we use discriminative models or generative models? Despite the fact that generative models have been frequently employed in a semi-supervised setting since the early days of the statistical revolution in NLP, we advocate the use of discriminative models. The ability of discriminative models to handle complex, high-dimensional feature spaces and their strong theoretical guarantees have made them a very appealing alternative to their generative counterparts. Perhaps more importantly, discriminative models have been shown to offer competitive performance on a variety of sequential and structured learning tasks in NLP that are traditionally tackled via generative models, such as letter-to-phoneme conversion (Jiampojamarn et al., 2008), semantic role labeling (Toutanova et al., 2005), syntactic parsing (Taskar et al., 2004), language modeling (Roark et al., 2004), and machine translation (Liang et al., 2006). While generative models allow the seamless integration of prior knowledge, discriminative models seem to outperform generative models in a no prior, agnostic learning setting. See Ng and Jordan (2002) and Toutanova (2006) for insightful comparisons of generative and discriminative models.

Computer vision can be understood as the ability to perform inference on image data. Breakthroughs in computer vision technology are often marked by advances in inference techniques. This thesis proposes novel inference schemes and demonstrates applications in computer vision. We propose inference techniques for both generative and discriminative vision models. The use of generative models in vision is often hampered by the difficulty of posterior inference. We propose techniques for improving inference in MCMC sampling and message-passing inference. Our inference strategy is to learn separate discriminative models that assist Bayesian inference in a generative model. Experiments on a range of generative models show that the proposed techniques accelerate the inference process and/or converge to better solutions. A main complication in the design of discriminative models is the inclusion of prior knowledge. We concentrate on CNN models and propose a generalization of standard spatial convolutions to bilateral convolutions. We generalize the existing use of bilateral filters and then propose new neural network architectures with learnable bilateral filters, which we call `Bilateral Neural Networks'. Experiments demonstrate the use of the bilateral networks on a wide range of image and video tasks and datasets. In summary, we propose techniques for better inference in several vision models ranging from inverse graphics to freely parameterized neural networks. In generative models, our inference techniques alleviate some of the crucial hurdles in Bayesian posterior inference, paving new ways for the use of model based machine learning in vision. In discriminative CNN models, the proposed filter generalizations aid in the design of new neural network architectures that can handle sparse high-dimensional data as well as provide a way to incorporate prior knowledge into CNNs.

Traditional Blind Source Separation Evaluation (BSS-Eval) metrics were originally designed to evaluate linear audio source separation models based on methods such as time-frequency masking. However, recent generative models may introduce nonlinear relationships between the separated and reference signals, limiting the reliability of these metrics for objective evaluation. To address this issue, we conduct a Degradation Category Rating listening test and analyze correlations between the obtained degradation mean opinion scores (DMOS) and a set of objective audio quality metrics for the task of singing voice separation. We evaluate three state-of-the-art discriminative models and two new, competitive generative models. For both discriminative and generative models, intrusive embedding-based metrics show higher correlations with DMOS than conventional intrusive metrics such as BSS-Eval metrics. For discriminative models, the highest correlation is achieved by the MSE computed on Music2Latent embeddings. When it comes to the evaluation of generative models, the strongest correlations are evident for the multi-resolution STFT loss and the MSE calculated on MERT-L12 embeddings, with the latter also providing the most balanced correlation across both model types. Our results highlight the limitations of BSS-Eval metrics for evaluating generative singing voice separation models and emphasize the need for careful selection and validation of alternative evaluation metrics for the task of singing voice separation.

RF fingerprint recognition is an emerging technology for identifying specific hardware features of wireless transmitters. In order to solve the problem of illegal transmitter hazardous wireless communication security, this paper proposes a method of generating RF fingerprint recognition methods based on generative adversarial net (GAN).This article first uses I / Q data through wavelet transform data pre-processing , Since the wavelet transform can describe the features of different frequency signals, the characteristics of radio frequency fingerprint can be highlighted after wavelet transform of I / Q data. Then we have designed a generative adversarial net model, which consists of a generate model and a discriminant model. Generate model input noise to generate a pseudo data distribution, The discriminant model enters the data distribution of the real trusted transmitter and the generated dummy data distribution generated, and the determination result is fed back to the generator, allowing the generator to update to generate more real pseudo data distributions, better The network model is used for radio frequency fingerprint recognition. Based on the above, it is possible to effectively identify rogue radio frequency transmitters to some extent to solve wireless security issues. The experimental results show that the generative adversarial net (GAN) can distinguish between 98.1% accuracy of the credibility transmitter and illegal transmitter, it has higher accuracy than traditional convolutional neural networks (CNN) and full connectivity neural network (DNN).

In this paper, we propose an effective way for combining the discriminative and generative models for emotion recognition from speech signal. Finding an efficient feature extraction algorithm which captures just the main attribute(s) pertinent to the task and filters out the other aspects of the data turns out to be very challenging, if not impossible. We propose an interface between the front-end and the back-end in order to compensate for the shortcoming of the parameterization block in suppressing the irrelevant dimensions of the signal. This interface is a generative model, which performs remarkable dimensionality reduction, allows for extraction of a long-term feature, and also paves the way for better classification of the data through a discriminative model. This method leads to a 7.6% absolute performance improvement in comparison with the baseline system and results in 87.6% accuracy in emotion recognition task. Human performance on the same database is reportedly 84.3%.

Comparative analysis of methods for cloud segmentation in ground-based infrared images

We aim to develop a conditional generative diffusion model capable of producing three-dimensional (3D) trabecular bone samples that can be tuned to achieve specific structural characteristics prescribed in terms of three geometric metrics of trabecular microarchitecture: bone volume fraction (BV/TV), trabecular thickness (Tb.Th), and spacing (Tb.Sp). The generative model is based on 3D latent diffusion. The latent representation of trabecular patches is obtained by a dedicated variational autoencoder (VAE). To control the microstructure characteristics of the synthetic samples, the model is conditioned on BV/TV, Tb.Th, and Tb.Sp. In addition, a shifting slab inference method is employed to generate extended volumes with locally tunable microstructure in a computationally efficient manner. The training data involved 3551 volumes of interest (VOIs) extracted from micro-CT volumes ( voxel size) of 20 femoral bone specimens, paired with trabecular metrics computed within each VOI; the split for training and validation data was 9:1. For testing, 2000 synthetic bone samples were generated using single slab inference over a wide range of condition (target) microstructure metrics. Results were evaluated in terms of (i)consistency across multiple realizations of reverse diffusion for a fixed condition, measured by the coefficient of variation (CV) of trabecular measurements; (ii)agreement between BV/TV, Tb.Th, and Tb.Sp values provided as a condition and those measured in the corresponding synthetic samples, assessed using Pearson correlation coefficient (PCC); and (iii)overlap between the distributions of trabecular parameters of real and synthetic bone patches; this coverage analysis included both the conditioning parameters of BV/TV, Tb.Th, and Tb.Sp, as well as unconditioned metrics of degree of anisotropy, ellipsoid factor, and connectivity. Further, extended volumes ( ) were generated using shifting-slab inference with spatially invariant and spatially varying conditioning and evaluated in terms of local agreement between the prescribed and achieved trabecular parameters. Visually, the synthesized cancellous bone patches appear highly similar to the training micro-CT data. The conditioned parameters of the generated volumes agree well with their target values (PCC of 0.99, 0.97, and 0.95 for BV/TV, Tb.Th, and Tb.Sp, respectively). There is a trend toward generating trabeculae that are slightly thicker than prescribed, but this bias is typically on the order of one voxel ( ). The metrics of BV/TV, Tb.Th, and Tb.Sp remain stable across multiple model inferences with a fixed condition (CV of ). Joint distributions of microstructure parameters of the synthetic samples capture the real-world distributions of the training data, with a slight underrepresentation of cases with large Tb.Sp ( ), attributed to imbalances in the training set. The shifting slab mechanism resulted in realistic and continuous trabecular structures with variable local architecture that accurately matched the prescribed spatial variation of the conditioned metrics of microstructure. The proposed generative model is capable of generating realistic digital trabecular bone patches. The application of latent space diffusion using a dedicated VAE augmented with the shifting-slab mechanism effectively overcomes computer memory constraints to enable the synthesis of 3D volumes. The conditioning mechanism is effective in guiding the synthesis toward desired microstructural characteristics. Possible applications include virtual clinical trials of new skeletal image biomarkers and establishing priors for advanced image reconstruction.