The decay of pulsed eddy currents (PEC) with depth in diffusion decreases resultant signal changes, thus bringing a challenge for reliable and accurate evaluation of deep subsurface defects. In this work, a novel cross-correlation inspired residual network, termed CCResNet, is proposed to improve the capability for smart evaluation of subsurface defects. It consists of a cross-correlation layer, a residual network, and a novel loss function, namely, focal-probability of detection (Focal-POD) loss. The customized Gaussian wavelet basis enables us to derive weak features from heavily noised PEC signals due to the similarity by cross-correlation operation, which is the origin of the constructed cross-correlation layer. Then, a Focal-POD loss is proposed to address class imbalance and endow CCResNet with powerful capability for detection of deep subsurface defects by increasing their loss values. Finally, a semi-supervised framework is built to re-train CCResNet using pseudo and labelled dataset to obtain classified results as imaging features. The experimental results show that the developed CCResNet is featured as better imaging resolution, more accurate evaluation, and intelligence in detection of deeper subsurface defects.