Solar radiation is the excitation source that affects the weather in the atmosphere of the earth, and some solar activities such as flares and coronal mass ejections are often accompanied by radio bursts. The spectrum of solar radio bursts is helpful for astronomers to explore the mechanism of radio bursts. With the development and progress of solar radio spectrum observation methods, the observation of the Sun can be done at almost all times of day. How to quickly and automatically identify the small proportion of burst data from the huge corpus of observation data has become an important research direction. The innovation of this study is to enhance the original radio spectrum dataset with unbalanced sample distribution, and a neural network model for solar radio spectrum image classification is proposed on this basis. This hybrid structure of joint convolution and a memory unit overcomes the shortcoming of the traditional convolution or memory model, which can only extract one-sided features of an image. By extracting the frequency structure features and time-series features at the same time, the sensitivity to the small features of the spectrum image can be enhanced. Based on the data of the Solar Broadband Radio Spectrometer (SBRS) in China, the proposed network model can improve the average classification accuracy of the spectrum image to 98.73%, which will be helpful for related astronomical research.