Refractory materials are basic materials widely used in industrial furnaces and thermal equipment. Their microstructure is similar to that of many heterogeneous high-performance materials used in micro/nanodevices. The presence of damage can reduce the mechanical properties and service life of refractory materials and even cause serious safety accidents. In this paper, a novel percussion and particle swarm optimization-support vector machine (PSO-SVM)-based method is proposed to detect damage in refractory materials. An impact is applied to the material and the generated sound is recorded. The percussion-induced sound signals are fed into a mel filter bank to generate time-frequency representations in the form of mel spectrograms. Then, two image descriptors-the local binary pattern (LBP) and histogram of oriented gradient (HOG)-are used to extract the texture information of the mel spectrogram. Finally, combining both HOG and LBP features, the fused features are input to the PSO-SVM algorithm to realize damage detection in refractory materials. The results demonstrated that the proposed method could identify five different degrees of damage of refractory materials, with an accuracy rate greater than 97%. Therefore, the percussion and PSO-SVM-based method proposed in this paper has high potential for field applications in damage detection in refractory material, and also has the potential to be extended to research on damage detection methods for other materials used in micro/nanodevices.