Optimizing and Evaluating Swin Transformer for Aircraft Classification: Analysis and Generalizability of the MTARSI Dataset

Abstract

Aircraft classification via remote sensing images has many commercial and military applications. The Swin-Transformer has shown great promise, recently dominating general purpose image classification benchmarks such as ImageNet. In this manuscript, we test whether the performance of Swin-Transformer on general purpose image classification translate to domain specific aircraft classification using the Multi-Type Aircraft from Remote Sensing Images dataset. We also investigate the effect of training procedure vs. model selection on the validation score. Our carefully trained Swin-Transformer model achieved an impressive 99.4 % validation set accuracy without super-resolution, and 99.5 % with super-resolution. Moreover, the generalization of models trained on the MTARSI dataset to real-world and synthetic aircraft classification is evaluated with some out-of-distribution samples. Our results demonstrated that the lack of complexity and heterogeneity of the MTARSI dataset, and the labelling errors resulted in models which struggle to achieve high accuracy on the adopted test samples despite near perfect validation scores.