Aflatoxin B1 (AFB1) is the most widely distributed, most toxic, and most harmful, and it is widely present in moldy grains. This study proposes a new method for quantitative and rapid determination of the AFB1 content in maize based on Raman spectroscopy. The Raman spectra of maize samples with different mildew degrees were collected by a portable laser Raman spectroscopy system. Three different spectral selection methods, which were bootstrapping soft shrinkage (BOSS), variable combination population analysis (VCPA) and competitive adaptive reweighted sampling (CARS), were applied to optimize the characteristic wavelength variables of the pretreated Raman spectra. The support vector machine (SVM) detection models based on different optimized characteristic wavelength variables were established, and the results of each detection model were compared. The results obtained showed that the performance of the SVM models established by optimized features was significantly better than the performance of the SVM model built by full-spectrum data. Among them, the SVM model based on the characteristic wavelength variables optimized by the CARS method had the best performance, and its root mean square error of prediction (RMSEP) was 3.5377 μg∙kg−1, the determination coefficient of prediction (RP2) was 0.9715, and the relative prediction deviation (RPD) was 5.8258. The overall results reveal that the rapid quantitative detection of the AFB1 in maize by Raman spectroscopy has a promising application prospect. In addition, the implementation of the characteristic wavelength optimization of Raman spectra in the model calibration process can effectively improve the detection accuracy of chemometric models.