Monitoring soil organic carbon (SOC) content is crucial for climate change mitigation and sustaining ecological balance. Despite the unparalleled advantages of hyperspectral data in capturing nuanced variations in soil properties through its high spectral resolution, effectively extracting useful features from numerous bands via spectral processing techniques remains a formidable challenge. This study proposes an integrated approach combining fractional-order derivative (FOD) technique and optimal band combination algorithm using ZY1-02D satellite hyperspectral data to estimate SOC in Northeast China’s Black soil region. Three modeling strategies were compared: (1) FOD-transformed reflectance (FOD spectra), (2) FOD spectra with traditional 2D spectral indices (FOD + 2D SI), and (3) FOD spectra with new 3D spectral indices (FOD + 3D SI). These strategies were implemented using the random forest model with the aim of the optimal SOC prediction. Results showed that the application of FOD technique for spectral transformation effectively addressed the challenges posed by overlapping peaks and baseline drift inherent in the original spectral reflectance. Additionally, FOD transformation enhanced subtle soil spectral features and yielded more pronounced spectral variations with increasing fractional order, as compared to the original spectral data and conventional integer-order derivatives (i.e., first and second-order derivatives). However, as the FOD order continued to increase beyond 1.4, the spectral curve exhibited amplified noise and distortion, thereby adversely impacting subsequent model development. The 3D spectral indices correlate more robustly with SOC than 2D indices. The model that combines 0.6-order FOD and 3D spectral indices achieved the best accuracy (R2 = 0.66, RMSE = 2.99 g/kg and MAE = 2.42 g/kg), significantly outperforming the models built by 0.6-order FOD spectra (R2 = 0.48, RMSE = 3.65 g kg−1, and MAE = 2.93 g kg−1) and 0.8-order FOD + 2D SI modeling strategy (R2 = 0.55, RMSE = 3.54 g kg−1, and MAE = 2.85 g kg−1). These findings indicated that FOD and 3D spectral indices exhibit superior synergistic performance in SOC prediction, demonstrating their feasibility and providing valuable insights for large-scale soil property prediction and mapping using satellite hyperspectral data.