Generative Adversarial Networks (GANs) demonstrate great success in content generation ranging from music to images. Classical GANs require an enormous amount of training data to match the distribution of real images, otherwise lead to overfitting and display implausible synthesis qualities. Previous methods adapt the pretrained model trained on large-scale data to the limited data but fail to handle the extreme few-shot cases. In this paper, we propose a self-supervised learning (SSL)-based regularization method to adapt the existing model to semantically related target domains. Our SSL-based regularization method constrains the relative distance between the source and adapting generator to transfer the diversity from the source domain to the target domain. To encourage a healthy competition with the generator, we also regularize the discriminator network to preserve the relative distances in the feature space. With extensive qualitative and quantitative evaluation on a variety of domains, our method demonstrates superior performance in terms of fidelity and diversity than existing state-of-the-art models. We achieve an FID score of 30.61 on the publicly available FFHQ Sunglasses dataset.