Automatic classification of heart sound plays an important role in the diagnosis of cardiovascular diseases. In this study, a heart sound sample classification method based on quality assessment and wavelet scattering transform was proposed. First, the ratio of zero crossings (RZC) and the root mean square of successive differences (RMSSD) were used for assessing the quality of heart sound signal. The first signal segment conforming to the threshold standard was selected as the current sample for the continuous heart sound signal. Using the wavelet scattering transform, the wavelet scattering coefficients were expanded according to the wavelet scale dimension, to obtain the features. Support vector machine (SVM) was used for classification, and the classification results for the samples were obtained using the wavelet scale dimension voting approach. The effects of RZC and RMSSD on the results are discussed in detail. On the database of PhysioNet Computing in Cardiology Challenge 2016 (CinC 2016), the proposed method yields 92.23% accuracy (Acc), 96.62% sensitivity (Se), 90.65% specificity (Sp), and 93.64% measure of accuracy (Macc). The results show that the proposed method can effectively classify normal and abnormal heart sound samples with high accuracy.