Learning a Prototype Discriminator With RBF for Multimodal Image Synthesis.

Abstract

Multimodal image synthesis has emerged as a viable solution to the modality missing challenge. Most existing approaches employ softmax-based classifiers to provide modal constraints for the generated models. These methods, however, focus on learning to distinguish inter-domain differences while failing to build intra-domain compactness, resulting in inferior synthetic results. To provide sufficient domain-specific constraint, we hereby introduce a novel prototype discriminator for generative adversarial network (PT-GAN) to effectively estimate the missing or noisy modalities. Different from most previous works, we introduce the Radial Basis Function (RBF) network, endowing the discriminator with domain-specific prototypes, to improve the optimization of generative model. Since the prototype learning extracts more discriminative representation of each domain, and emphasizes intra-domain compactness, it reduces the sensitivity of discriminator to pixel changes in generated images. To address this dilemma, we further propose a reconstructive regularization term which connects the discriminator with the generator, thus enhancing its pixel detectability. To this end, the proposed PT-GAN provides not only consistent domain-specific constraints, but also reasonable uncertainty estimation of generated images with the RBF distance. Experimental results show that our method outperforms the state-of-the-art techniques. The source code will be available at: https://github.com/zhiweibi/PT-GAN.