BTN: Neuroanatomical aligning between visual object tracking in deep neural network and smooth pursuit in brain

Abstract

Inspired by neuroanatomy, deep neural networks (DNNs) have recently developed from shallow network structures to exceedingly deep structures, providing excellent visual tracking results. However, standard DNNs usually do not easily connect with brain areas on account of their excessive network depth and absent biological constraints, such as recurrent connections. We propose a brain-like tracking network (BTN) with four neuroanatomically mapped regions and recurrence, guided by the brain-like tracking score (BTS), a novel benchmark to measure the model similarity of the human smooth pursuit pathway. In addition, we propose that the middle temporal (MT) and medial superior temporal (MST) areas in the cerebral cortex are equivalent to the designed network structure on the basis of the continuous motion perception of the tracking pathway, and indicate that the metrics between the neuroanatomical similarity in the cerebral cortex and visual tracking of the DNN are compatible. Despite having significant tracking performance on the Tracking-Gump dataset, the BTN has achieved a high BTS. In summary, this research builds a BTN, a brain-like and recurrent DNN, as the first model of the cortical pathway of smooth pursuit.