Exploration of classical neural network architecture in cycleGAN framework with face photo-sketch synthesis

Abstract

CycleGAN has been a benchmark in the style transfer field and various extensions with wide applications and excellent performance have been introduced in recent years, however, discussion about its architecture exploration which could enable us to further understand the concept of generative model is scarce. In this paper, several architectures referenced from classical convolutional neural networks are implemented into the generator and discriminator of the cycleGAN model, including AlexNet, DenseNet, GoogLeNet, and ResNet. Their feature extraction modes are imitated and modified into blocks to embed into the encoder part of the generator while the discriminator directly uses their model except it outputs a patch classification. In advance to mitigate the possible imbalance between generator and discriminator ability, a self-adjusting learning rate strategy based on the discriminator confidence is introduced. Multiple evaluation metrics are utilized to measure the performance of each model. Experimental results indicate an AlexNet-like architecture model could achieve a competitive performance than the baseline cycleGAN and present better fine details and high-frequency information.