Speech plays a major role in emotional transmitting information in humans, and speech emotion recognition has become an important part of the human–computer system, especially in specific systems with high requirements for real-time and accuracy. To improve the accuracy and real-time of speech emotion recognition, people have done a lot of work in speech emotion feature extraction and speech emotion recognition algorithms, but the recognition rate also needs improvement. In this paper, we propose a speech emotion recognition method based on Mel-frequency Cepstral coefficients (MFCC) and broad learning network. 39-dimensional MFCC features were extracted after preprocess of the speech signal. After labelling and standardizing the data, a data prediction model is built. Finally, the data set is split into training and test data onto a certain ratio (0.8). We experimented with broad learning network architecture. And then the data processing in the broad learning network is improved. The proposed algorithm is a neural network structure that does not rely on deep structure, which has a small amount of calculation, excellent calculation speed and simple structure. The experimental results show that the proposed network architecture achieves higher accuracy and it turned out to be the most accurate in recognizing emotions in CASIA Chinese emotion corpus. The recognition rate can reach 100%. Therefore, the proposed network architecture provides an effective method of speech emotion recognition.