Pairwise Cluster Similarity Domain Adaptation for Multimodal Deep Learning Architecture

Abstract

In this work, we investigate the drawback of direct application from a dataset to the new domain with a different probability distribution, known as domain shifting. The two different probability distribution of datasets is generally known as source domain and target domain. Regularly, when we use large datasets from the open repository (source domain) for the new problem (target domain), we need to bridge for adapting these two domains. Notably, this research focuses on learning setup for the multimodal task which is commonly known as domain adaptation. To minimize the difference between the source and target domain, we employ cluster analysis with a modification of the loss function of a deep neural network. The modification of the loss function is done by combining the cross-entropy with covariance between source and target domains. We propose a method named Pairwise Cluster Similarity Domain Adaptation (PCSDA) that orders the distance among clusters from the source and target domains. Comprehensive experimental results of image captioning task (object detection on image and text generation) demonstrate the robustness and effectiveness of the proposed method.