Lipid oxidation is the main cause of meat deterioration. Hyperspectral imaging (HSI) technique has attracted attention as a non-destructive testing method. However, the complexity and overlap of the pork hyperspectral data lead to difficult band interpretation and computational overload. In this paper, a lightweight three-dimensional convolutional neural network (3D-CNN) model combined with two-dimensional correlation spectroscopy (2D-COS) analysis was proposed to monitor the lipid oxidation of frozen pork. Through the generalized 2D-COS analysis, the band interpretation of visible near-infrared (vis-NIR) HSI was established and the sequence of event changes caused by pork deterioration was monitored. It was found that sulfmyoglobin and oxymyoglobin were prone to change, and the decomposition of sulfmyoglobin and metmyoglobin occurred before the formation of oxymyoglobin. Moreover, the hetero 2D-COS analysis was used for the first time to correlate vis-NIR with fluorescence spectra to analyze more feature bands of vis-NIR HSI. A lightweight 3D-CNN regression model was developed for hyperspectral images of feature bands to quantitatively predict TBARS. It was found that 10 feature bands were obtained by integrating bands identified by generalized and hetero 2D-COS. The 3D-CNN model of these feature bands has yielded good results in predicting TBARS with R2p of 0.9214 and RMSEP of 0.0364 mg kg−1. Overall, this study provided a method for band assignment and interpretation of vis-NIR HSI and an end-to-end new approach for rapid and non-destructive monitoring of pork oxidative damage.