In this study, an innovative dual-aspect attention spatial-spectral transformer (DAASST) was introduced to advance postharvest quality control by the detection of Aspergillus flavus contamination and the accurate identification of contamination times in peanut kernels. The critical importance of maintaining postharvest quality and safety in nuts was recognized, with hyperspectral imaging technology being leveraged due to its great potential in non-destructive testing and quality assessment of nuts. At the heart of DAASST's innovation, an enhanced transformer architecture that incorporated an attention fusion mechanism was employed for the effective integration of the extracted features. This sophisticated integration not only improved the model’s performance but also was significantly surpassed by the capabilities of traditional machine learning methods in the context of postharvest biology and technology. Exceptional accuracy was demonstrated in testing, with 99.40% achieved in detecting Aspergillus flavus contamination and a remarkable 100% in distinguishing between different contamination times. Significant contributions to the field of postharvest biology and technology were made by merging cutting-edge feature extraction techniques, attention mechanisms, and transformer architecture to refine hyperspectral image analysis for postharvest quality control. The proven effectiveness of the DAASST in accurately detecting Aspergillus flavus and determining contamination times in peanut kernels highlighted its potential as a valuable tool for ensuring the safety and quality of postharvest nuts.