Accurate classification of sorghum varieties is crucial to the production and processing of liquor with sorghum as raw materials. Hyperspectral imaging (HSI) has the potential to achieve this goal quickly and nondestructively. This study proposes a novel algorithm, an improved Principal Component Analysis combined with Spectrum-Image-Convolutional Neural Network (PCA-SICNN), which can combine spectral features and image features of HSI data, to enhance the accuracy of variety identification of sorghum seeds. To verify the effect of this algorithm, hyperspectral imaging data (939–1700 nm) of 13,200 sorghum seeds from 6 varieties were collected. The principal component analysis (PCA) was employed to select 20-dimension images from the origin hyperspectral imaging data. Spectrum-Image-Convolutional Neural Network (SICNN) extracts the spectral and image features of sorghum in the network and then fuses the features. By fully learning the HSI data features of sorghum, it achieves the classification of sorghum varieties. The results demonstrate that PCA-SICNN achieves an accuracy on the training set and on the test set reaches 98.67 % and 98.64 %, respectively. Compared with other control methods, the prediction accuracy of the PCA-SICNN model increased by at least 1.10 %. These results suggest the potential for the method to be widely applied in the production and processing of sorghum.