TFMFT: Transformer-based multiple fish tracking

Abstract

Recent advancements in fish tracking methodologies provide valuable solutions for assessing fish growth, marine fisheries, and biological research. In particular, there has been a burgeoning interest in vision-based methods for fish tracking, owing to the enhanced computational capabilities facilitated by deep learning models. However, these methods face several challenges, including poor fish detection performance under complex backgrounds, the potential for identification switches caused by the non-rigid features and occlusions of fish, and the limited fault tolerance of extant approaches. In this paper, a transformer-based multiple fish tracking model (TFMFT) is proposed, specifically designed to address the issue of instance loss of fish targets in aquaculture ponds with complex background disturbance. In particular, we introduce a Multiple Association (MA) method that bolsters fault tolerance in tracking by synthesizing simple Intersection-over-Union matching in the identification (ID) matching module. Through empirical studies across diverse Transformer-based models, we comprehensively assessed the influence of architecture design on data requirements. Furthermore, to evaluate the performance and generalizability of fish tracking models, we present the Multiple_Fish_Tracking_2022 (MFT22) dataset. The results demonstrate that TFMFT achieves 30.6% IDF1 (Identification F-Score) on the MFT22 dataset, outperforming the state-of-the-art by 10.9% and showcasing superior performance over other models. The resources and pre-trained model will be available at: https://github.com/vranlee/TFMFT.