The application of hyperspectral imaging with computer-aided technology has promising prospects, and achieving real-time, efficient, and non-destructive detection, especially for food and agricultural products, undoubtedly poses a great challenge. Hyperspectral data processing has many complications, such as large volume, high redundancy, and difficulty extracting useful features. Therefore, this study develops a lightweight end-to-end unified framework for deep neural networks with excellent generalization to conserve memory space and computation. To improve the classification accuracy and performance of the core model LSAC-net, we combine an attention mechanism based on the temporal convolution method with a complementary model-scaling technique. Experimental results on our own datasets for the production year of citri reticulate pericarpium, production origin of Pu’er tea, and process mode of coffee beans show that our model outperforms several other state-of-the-art models.