ABSTRACT The excavated areas formed by human activities often indicate the destruction of the ecological environment and topography, so it is of great significance to carry out dynamic monitoring. For the characteristics of the complex structure, different spatial scales, and less vegetation coverage in the excavated area, this study proposes a hierarchical spatial pyramid pooling (H-SPP) structure to effectively obtain the multiscale complex structural features from remote sensing images. Then, we add the H-SPP structure to the VGG-16 network to form a multiscale visual geometry group (M-VGG) suitable for excavated area detection and optimize the network by using the average pooling method to obtain more global information. The vegetation index is used to mask the background information. The experimental results show that the proposed method significantly improves the object detection performance for the excavated area compared with other methods. The detection time is reduced by almost 90%, and the accuracy is increased by more than 4%.