The non-destructive testing of litchi fruit is of great significance to the fresh-keeping, storage and transportation of harvested litchis. To achieve quick and accurate micro-damage detection, a non-destructive grading test method for litchi fruits was studied using 400–1000 nm hyperspectral imaging technology. The Huaizhi litchi was chosen in this study, and the hyperspectral data average for the region of interest (ROI) of litchi fruit was extracted for spectral data analysis. Then the hyperspectral data samples of fresh and micro-damaged litchi fruits were selected, and a partial least squares discriminant analysis (PLS-DA) was used to establish a prediction model for the realization of qualitative analysis for litchis with different qualities. For the external validation set, the mean per-type recall and precision were 94.10% and 93.95%, respectively. Principal component analysis (PCA) was used to determine the sensitive wavelength for recognition of litchi quality characteristics, with the results of wavelengths corresponding to the local extremum for the weight coefficient of PC3, i.e., 694, 725 and 798 nm. Then the single-band images corresponding to each sensitive wavelength were analyzed. Finally, the 7-dimension features of the PC3 image were extracted using the Gray Level Co-occurrence Matrix (GLCM). Through image processing, least squares support vector machine (LS-SVM) modeling was conducted to classify the different qualities of litchis. The model was validated using the experiment data, and the average accuracy of the validation set was 93.75%, while the external validation set was 95%. The results indicate the feasibility of using hyperspectral imaging technology in litchi postpartum non-destructive detection and classification.