Policy Transfer from Simulations to Real World by Transfer Component Analysis

Abstract

Learning of control policies that map visual observations to actions has been drawing much attention in recent years. Since visual observations are defined as raw images or high-dimensional image features, the learning process of such policies tends to require a huge amount of training data. Therefore, their applications are often limited in simulations and difficult in the real world due to the high sampling cost of the training data. To overcome this limitation, in this paper, we propose a policy transfer learning framework with feature extraction and reinforcement learning. We utilize transfer component analysis as a feature extraction method for identifying common features across the two domains. Our framework first extracts the common features in both the simulations and the real environments. Then a control policy with the extracted features as observations is learned by reinforcement learning in simulations and subsequently transferred to the real world even with appearance errors. To validate the effectiveness of our framework, we conducted simulations and experiments with an actual robot platform.