The inherent non-uniformity generated in infrared focal plane arrays (IRFPA) due to inconsistent responses of detectors, and it significantly degrades the performance of infrared imaging systems. Traditional Non-Uniformity Correction (NUC) methods mainly leverage the temporal low-frequency property of non-uniformity to remove it. In recent years, Convolutional Neural Network (CNN) has gained attention as a powerful tool for conducting NUC using solely spatial information from a single frame. In this study, we propose a deep neural network with a recurrent spatio-temporal feature fusion module to enhance CNN performance using inter-frame information. Initially, the proposed network uses CNN to coarsely extract FPN features solely from the spatial domain. Subsequently, the network merges it with the historical estimation under the guidance of two gating signals, which calculated from spatio-temporal information with gated convolutional layers. Both stages are jointly trained to ensure that the entire network operates end-to-end and adapts to different scenarios, eliminating complex parameter tuning efforts. Experiments demonstrated that the proposed network shows both the advantages of traditional and CNN-based NUC methods. It has a higher convergence starting point, faster convergence speed, and a 1 ∼ 2 dB improvement in PSNR compared to both types of methods.