Soil visible and near-infrared reflectance spectroscopy is an effective tool for the rapid estimation of soil organic carbon (SOC). The development of spectroscopic technology has increased the application of spectral libraries for SOC research. However, the direct application of spectral libraries for SOC prediction remains challenging due to the high variability in soil types and soil-forming factors. This study aims to address this challenge by improving SOC prediction accuracy through spectral classification. We utilized the European Land Use and Cover Area frame Survey (LUCAS) large-scale spectral library and employed a geographically weighted principal component analysis (GWPCA) combined with a fuzzy c-means (FCM) clustering algorithm to classify the spectra. Subsequently, we used partial least squares regression (PLSR) and the Cubist model for SOC prediction. Additionally, we classified the soil data by land cover types and compared the classification prediction results with those obtained from spectral classification. The results showed that (1) the GWPCA-FCM-Cubist model yielded the best predictions, with an average accuracy of R2 = 0.83 and RPIQ = 2.95, representing improvements of 10.33% and 18.00% in R2 and RPIQ, respectively, compared to unclassified full sample modeling. (2) The accuracy of spectral classification modeling based on GWPCA-FCM was significantly superior to that of land cover type classification modeling. Specifically, there was a 7.64% and 14.22% improvement in R2 and RPIQ, respectively, under PLSR, and a 13.36% and 29.10% improvement in R2 and RPIQ, respectively, under Cubist. (3) Overall, the prediction accuracy of Cubist models was better than that of PLSR models. These findings indicate that the application of GWPCA and FCM clustering in conjunction with the Cubist modeling technique can significantly enhance the prediction accuracy of SOC from large-scale spectral libraries.