Pulsed eddy current thermography can detect surface or subsurface defects in steel, but in the process of combining deep learning, it is expensive and inefficient to build a complete sample of defects due to the complexity of the actual industrial environment. Consequently, this study proposes a transfer learning method based on Twin-NMF and combines it with the SimAM attention mechanism to enhance the detection accuracy of the target domain task. First, to address the domain differences between the target domain task and the source domain samples, this study introduces a Twin-NMF transfer method. This approach reconstructs the feature space of both the source and target domains using twin non-negative matrix factorization and employs cosine similarity to measure the correlation between the features of these two domains. Secondly, this study integrates a parameter-free SimAM into the neck of the YOLOv8 model to enhance its capabilities in extracting and classifying steel surface defects, as well as to alleviate the precision collapse phenomenon associated with multi-scale defect recognition. The experimental results show that the proposed Twin-NMF model with SimAM improves the detection accuracy of steel surface defects. Taking NEU-DET and GC10-DET as source domains, respectively, in the ECTI dataset, mAP@0.5 reaches 99.3% and 99.2%, and the detection accuracy reaches 98% and 98.5%.