Though soil is widely known as one of the most valuable resources for the world, its quality is going to be lower because of unsustainable economic development and social progress. Therefore, it is important for us to monitor and evaluate the quality of soil, especially its heavy metal contents which is too scarce to identify in soil spectra easily but poisonous enough to affect human health in a long run. Most of the existing estimation methods have based the characteristic bands on statistical analysis to a large extent, which is hard to accurately explain the retrieval mechanism. In this paper, the absorption characteristics of heavy metal are studied based on the soil spectra, and the distribution pattern is mapped in a large-scale continuous space, for environmental monitoring and further decision support. Taking Yitong County, China as the study area. After spectra continuum removal, the heavy metal contents were estimated by 11 features including the absorption depth, absorption area, and band ratio around 2200 nm, which showed the best performance. For arsenic (As), the best model yields Rp2 value of 0.8474, and the RMSEP value is 36.1542 (mg/kg). It is concluded that As is adsorbed by organic matter, clay minerals, and iron/manganese oxides in soil, and the adsorption of As by first two components is greater than that of the last. For airborne spectra after continuum removal, combining the spectral absorption characteristic parameters and the highly correlated bands is more accurate than using the spectral absorption characteristic parameters or bands alone. AdaBoost is presented for the heavy metal estimation, and the fitting ability of the method is found to be stronger than that of the traditional classical methods, with the Rp2 values of 0.6242 and the RMSEP value of 43.6481 (mg/kg). In summary, these results will provide a prospective basis for the rapid estimation of soil heavy metals, the risk assessment of soil heavy metals and soil environmental monitoring in a large scale.