Infrared small target tracking plays important roles in many applications, such as infrared surveillance and reconnaissance. However, achieving promising performance with existing trackers is difficult due to the following challenges: (1) Infrared small targets lack color and texture information, making it difficult for the network to capture effective features; (2) A slight centroid-shift of the target can lead to tracking failure. To overcome these challenges, this paper proposes the CFNLA network, which consists of two parts: (1) The cross-fusion module is designed to capture the relationship between the target and its surroundings, including a non-local block to enhance the feature information of the target at the current scale and reduce interference such as background; (2) The confidence loss function is proposed to improve tracking stability by forcing the network to concentrate on centroid shift. Furthermore, a self-designed infrared small target tracking dataset, IRSTT, is constructed to evaluate the proposed algorithm. Finally, the experimental results on various open-source datasets and the IRSTT dataset verify the superior performance of the CFNLA network.