Fruits have become essential in people's daily lives, with both their external defects and internal quality receiving close attention from sellers and consumers alike. This study employs hyperspectral imaging technology combined with deep learning to rapidly and non-destructively detect external defects and soluble solids content (SSC) in Nanfeng mandarins. Hyperspectral data (380–1030 nm) were collected from Nanfeng mandarins with four types of defects (anthracnose, black spot, decay, and scarring) and sound fruits. Firstly, an end-to-end convolutional neural network (CNN) model for qualitative analysis was proposed, and its classification performance was compared with traditional classification models. Three preprocessing methods and three feature selection techniques were applied. The results showed that the CNN model based on competitive adaptive reweighting sampling (CARS) achieved the highest overall accuracy for defect discrimination (97.27%). Additionally, using 150 sound Nanfeng mandarins as subjects, quantitative predictive models for SSC were developed using full spectrum and feature wavelength-based partial least squares regression (PLSR), least squares support vector machine (LSSVM), and CNN. Among these, the best predictive model for the SSC of Nanfeng mandarins was the CNN, with R2, RMSEP, and RPD values of 0.9290, 0.3772, and 3.7655, respectively. Overall, this study has demonstrated the feasibility of using hyperspectral imaging combined with deep learning for defect identification and SSC prediction in Nanfeng mandarins, providing a new method for the internal and external quality assessment of other fruits.