Mud pulse telemetry is the most commonly used communication technology in measurement-while-drilling or logging-while-drilling systems. During the drilling process, the flowing drilling mud is the communication channel that carries the mud pulse signal to surface. Compared with the relatively benign conditions in the wireline communication method, the mud communication channel encounters high signal attenuation, low bandwidth, and numerous sources of noise. The harsh and volatile conditions of the mud channel make it difficult to demodulate the mud pulse signal. In this study, a novel signal demodulation technique based on support vector machines (SVM) is proposed. Meanwhile, the particle swarm optimization (PSO) algorithm is applied for selecting both the penalty parameter and the kernel parameter for the SVM classifier. To overcome the small training set problem and increase the demodulation accuracy, we introduce a training set expansion scheme which is implemented by enlarging the training set with correctly demodulated data based on the error detection method. Field experiment results prove the effectiveness of the training set expansion scheme and the superiority of the proposed methodology compared with the conventional adaptive fractionally-spaced decision-feedback equalizer (FS-DFE). Moreover, the computational time for the whole SVM demodulation procedure is much less than the signal duration. The proposed PSO–SVM method could be a potential solution for mud pulse signal demodulation in mud pulse telemetry systems.