To achieve the rapid grade classification of camellia seed oil, hyperspectral imaging technology was used to acquire hyperspectral images of three distinct grades of camellia seed oil. The spectral and image information collected by the hyperspectral imaging technology was preprocessed by different methods. The characteristic wavelength selection in this study included the continuous projections algorithm (SPA) and competitive adaptive reweighted sampling (CARS), and the gray-level co-occurrence matrix (GLCM) algorithm was used to extract the texture features of camellia seed oil at the characteristic wavelength. Combined with genetic algorithm (GA) and support vector machine algorithm (SVM), different grade classification models for camellia seed oil were developed using full wavelengths (GA-SVM), characteristic wavelengths (CARS-GA-SVM), and fusing spectral and image features (CARS-GLCM-GA-SVM). The results show that the CARS-GLCM-GA-SVM model, which combined spectral and image information, had the best classification effect, and the accuracy of the calibration set and prediction set of the CARS-GLCM-GA-SVM model were 98.30% and 96.61%, respectively. Compared with the CARS-GA-SVM model, the accuracy of the calibration set and prediction set were improved by 10.75% and 12.04%, respectively. Compared with the GA-SVM model, the accuracy of the calibration set and prediction set were improved by 18.28% and 18.15%, respectively. The research showed that hyperspectral imaging technology can rapidly classify camellia seed oil grades.