Defects or damages on the surface or subsurface of composite rolls directly affect the quality of the rolled products, and their periodic inspection and accurate identification can provide a reference for the repair or replacement of composite rolls. Ultrasonic Rayleigh waves can non-destructively assess the surface or subsurface of composite rolls, during which similar original signals are perhaps generated for the defects or damages with different types or sizes, making it difficult to distinguish their types. Machine learning is proposed to solve these problems, and feature selection based on the ultrasonic Rayleigh waves mainly utilizes traditional methods such as ReliefF. However, these unsupervised methods have not been combined with specific classification algorithms. In this study, swarm intelligence optimization algorithms, including snake optimizer (SO), dung beetle optimizer (DBO), and grasshopper optimization algorithm (GOA), are investigated based on the support vector machine (SVM). Note that the utilized original signals are received on the left and right sides of defects or damages, and they are processed by segmentation, fast Fourier transform (FFT), and wavelet packet decomposition to build original feature sets. It is illustrated that Rayleigh waves passing through the defects or damages carry more valuable information about the types. In contrast, the ultrasonic waves reflected from the defects or damages can provide information about the types that the former does not contain. Furthermore, for the intelligent classification of defects or damages in composite rolls using ultrasonic Rayleigh waves, SO is more suitable for SVM and has certain advantages.