GFSNet: Generalization-friendly siamese network for thermal infrared object tracking

Abstract

The lack of large labeled training datasets hinders the usage of deep neural network for Thermal Infrared (TIR) tracking. Regular practice is to train a tracking network with large-scale RGB datasets and then retrain it to the TIR domain with limited TIR data. However, we observe that existing Siamese-based trackers can hardly generalize to TIR images though they achieve outstanding performance on RGB tracking. Therefore, the main challenge is the generalization problem: How to design a generalization-friendly Siamese tracking network and what affects the network generalization. To tackle this problem, we introduce the self-adaption structure into Siamese network and propose an effective TIR tracking model, GFSNet. GFSNet is successfully generalized to different TIR tracking tasks, including ground target, aircraft and high-diversity object tracking tasks. To estimate generalization ability, we present a notion of Growth Rate, the improvement of overall performance after retraining. Experimental results show that the Growth Rates of GFSNet exceed state-of-the-art SiamRPN++ by more than 7 times, which indicates the great power of GFSNet in generalization. In addition to experimental validations, we provide the theoretical analysis of network generalization from a novel perspective, model sensitivity. By performing some tests to analyze the sensitivity, we conclude that the self-adaption structure helps GFSNet converge to a more sensitive minimum with better generalization to new tasks. Furthermore, when compared with popular tracking methods, GFSNet maintains comparable accuracy while achieving real-time tracking with the speed of 112 FPS, 5 times faster than other TIR trackers.