The stepped eddy current thermography nondestructive testing technique has a low and uneven temperature rise under weak excitation conditions, which brings great challenges to the quantitative analysis of defects in the inner wall of pressure vessels. To solve the above problems, this paper proposes a multi-spectral channel attention residual network model based on a Gramian angular difference field (GADF) for automatic detection and quantification. Firstly, the noise reduction and compression of temperature time series data are accomplished by singular value decomposition and piecewise aggregate approximation. Secondly, three infrared feature strategies commonly used for quantitative analysis were transformed by GADF and Gramian angular sum field (GASF) techniques to filter out the infrared feature data sets that contain the most defective feature information. Finally, the frequency domainization of the feature map is realized by embedding the multispectral channel attention in the residual network. The global dependencies and relevant essential features of infrared data are effectively captured, and the redundant feature information generated by deep learning feature maps is suppressed. The effectiveness of the model is verified by comparing several time-series models with advanced deep-learning models. The results show that the proposed model has excellent performance in terms of accuracy, stability, and generalization ability. Meanwhile, the model can effectively extract the defects in the region of uneven temperature rise.