Automatic layer selection for transfer learning and quantitative evaluation of layer effectiveness

Abstract

The performance of transfer learning in convolutional neural networks depends on the selection of which layer to update and fix. Because the number of layers is increasing, it is becoming increasingly difficult for humans to select layers. Therefore, in this study, we propose a method to automatically select effective update layers for transfer learning using a genetic algorithm. In our experiments, we conducted transfer learning from InceptionV3 pretrained with ImageNet to Canadian Institute for Advanced Research-100 dataset, The Street View House Numbers dataset and Food-101 dataset. We found that the test accuracy obtained by an ensemble of models selected by the genetic algorithm was greater than that obtained by from-scratch and fine-tuning for all target dataset. The distribution of the layers selected by the genetic algorithm as effective update layers was spread over the entire network. We also employed the optimal transport distance to evaluate whether each convolutional layer is an effective update layer for transfer learning. In our experiments, we compared the layer importance values and the accuracy of transfer learning. The layer importance was then correlated with the test accuracy of transfer learning, and the results demonstrate that the proposed method can quantitatively evaluate how well each network layer can detect general features in the target datasets.