Soil or sediment spectral reflectance do not directly reflect information of spectrally featureless heavy metals. With mangrove sediment samples collected in five regions and four habitat types, this study aimed to explore the spectral features related to spectrally active constituents (Fe and organic carbon (OC)) in estimating low and moderate concentrations of four heavy metals (Cr, Ni, Cu and Pb). A total of 27 modeling strategies using XGBoost (extreme gradient boosting) model were compared for estimating each heavy metal: three types of spectral features (Fe, OC and heavy metal) * three spectral transformation methods (Log(1/R), first derivative (FD) and continuous wavelet transform (CWT)) * three variable selection methods (genetic algorithm (GA), successive projection algorithm (SPA) and recursive feature elimination (RFE)). The results showed that the combination of CWT and RFE was recommended for heavy metal estimation, and the ranking of model accuracy was Cu > Pb > Ni > Cr. The optimal XGBoost model was found using spectral features related to Fe for Cr (RVal2(determination coefficient of independent validation) = 0.42, RMSE (root mean square error) = 0.32 and RPIQ (the ratio of performance to interquartile range) = 1.53) and Cu estimation (RVal2 = 0.87, RMSE = 0.19 and RPIQ = 2.20), spectral features related to OC for Ni estimation (RVal2 = 0.63, RMSE = 0.26 and RPIQ = 2.38) and spectral features related to Pb itself for Pb estimation (RVal2 = 0.82, RMSE = 0.23 and RPIQ = 2.59). Moreover, the XGBoost model transferability across different habitat types outperformed that across different regions and data ranges of Fe and OC. We conclude that the modeling strategy using spectral features related to spectrally active constituents has great potentials in quantifying low and moderate concentration of sediment heavy metals, and the samples with high Fe concentration (31.62–37.29 g kg−1) are recommended to improve estimation accuracy of low and moderate concentrations of heavy metals, which could provide sampling strategy development and early warning of heavy metal pollution in mangrove ecosystem.