A new feature extraction technique called DNST-GLCM-KSR (discrete non-separable shearlet transform-gray-level co-occurrence matrix-kernel spectral regression) is presented according to the direction and texture information of surface defects of continuous casting slabs with complex backgrounds. The discrete non-separable shearlet transform (DNST) is a new multi-scale geometric analysis method that provides excellent localization properties and directional selectivity. The gray-level co-occurrence matrix (GLCM) is a texture feature extraction technology. We combine DNST features with GLCM features to characterize defects of the continuous casting slabs. Since the combination feature is high-dimensional and redundant, kernel spectral regression (KSR) algorithm was used to remove redundancy. The low-dimension features obtained and labels data were inputted to a support vector machine (SVM) for classification. The samples collected from the continuous casting slab industrial production line—including cracks, scales, lighting variation, and slag marks—and the proposed scheme were tested. The test results show that the scheme can improve the classification accuracy to 96.37%, which provides a new approach for surface defect recognition of continuous casting slabs.