The building extraction from synthetic aperture radar (SAR) images has always been a challenging research topic. Recently, the deep convolution neural network brings excellent improvements in SAR segmentation. The fully convolutional network and other variants are widely transferred to the SAR studies because of their high precision in optical images. They are still limited by their processing in terms of the geometric distortion of buildings, the variability of building structures, and scattering interference between adjacent targets in the SAR images. In this article, a unified framework called selective spatial pyramid dilated (SSPD) network is proposed for the fine building segmentation in SAR images. First, we propose a novel encoder–decoder structure for the fine building feature reconstruction. The enhanced encoder and the dual-stage decoder, composed of the CBM and the SSPD module, extract and recover the crucial multiscale information better. Second, we design the multilayer SSPD module based on the selective spatial attention. The multiscale building information with different attention on multiple branches is combined, optimized, and adaptively selected for adaptive filtering and extracting features of complex multiscale building targets in SAR images. Third, according to the building features and SAR imaging mechanism, a new loss function called L-shape weighting loss (LWloss) is proposed to heighten the attention on the L-shape footprint characteristics of the buildings and reduce the missing detection of line buildings. Besides, LWloss can also alleviate the class imbalance problem in the optimization stage. Finally, the experiments on a large-scene SAR image dataset demonstrate the effectiveness of the proposed method and verify its superiority over other approaches, such as the region-based Markov random field, U-net, and DeepLabv3+.