Chrysanthemum, a widely favored flower tea, contains numerous phytochemicals for health benefits. Due to the different geographical origins and processing technics, its variety has a direct influence on the phytochemical content and pharmacological effect. Accordingly, an accurate identification for chrysanthemum varieties is significant for quality detection and market supervision. In this study, the hyperspectral imaging (HSI) combined with chemometrics methods was exploited to identify the chrysanthemum varieties. First, to alleviate the problem of easily trapping into local optimum in traditional spectral variable selection methods, the multi-tasking particle swarm optimization (MTPSO) was developed to select the key wavelengths by dividing hundreds of variables into low-dimensional subtasks. Second, to enrich the feature information, the spatial texture and color features contained in hyperspectral images were extracted and applied to chrysanthemum identification for the first time. Finally, an ensemble learning model, extreme gradient boosting (XGBoost), was constructed to conduct the chrysanthemum variety classification due to its strong generalization ability. Experimental results showed that the proposed MTPSO achieved the identification accuracy of 96.89%, and increased by 1.11–5.91% than classical spectral feature selection methods. Furthermore, after the involvement of spatial image information, the classification accuracy using spatial–spectral features was improved further, and reached 98.39%. Overall, this study highlights that the feature fusion of key wavelengths and spatial information is more effective for chrysanthemum variety identification, and can also provide technical reference for other HSI-related applications.