Analyzing Pre-Trained Neural Network Behavior with Layer Activation Optimization

Abstract

Image classification and object recognition with neural networks could have applications in aesthetically-focused branches of the humanities, such as landscape architecture. However, such methods require either the assembly of a massive, domain specific labeled data set or use of network weights initialized on another data set, a technique known as transfer learning. Transfer learning research has established that a pre-trained convolutional neural network (CNN) can achieve high accuracy on new image recognition tasks with relatively few training images. In practice, pre-trained tends to mean pre-trained on ImageNet, the standard dataset for computer vision research. Experiments have shown that the dataset on which a pre-trained model was originally optimized can quantitatively bias it. The goal of this project was to design an experiment to qualitatively analyze how the dataset used to initialize a pre-trained classification system affects its behavior at progressive network layers using feature visualization strategies. We initialized two ResNet-18 CNNs with weights pre-trained on ImageNet and the Places365 dataset, respectively, and fine-tuned them for a new classification task on a landscape image dataset which we collected. Using class activation optimization methods taken from the deep visualization literature, we compared the network filters at several hidden layers and the final output layers. The class activation optimization results show that even at early stages in the networks, their neurons exhibit notably different behavior. Accordingly, we show both that feature visualization techniques can be used to qualitatively study the effect of original training data on transfer learning and, consequently, that the homogeneous use of ImageNet in computer vision experiments may have notable implications for model behavior.