With the rapid development of very-high-resolution (VHR) remote-sensing technology, automatic identification and extraction of building footprints are significant for tracking urban development and evolution. Nevertheless, while VHR can more accurately characterize the details of buildings, it also inevitably enhances the background interference and noise information, which degrades the fine-grained detection of building footprints. In order to tackle the above issues, the attention mechanism is intensively exploited to provide a feasible solution. The attention mechanism is a computational intelligence technique inspired by the biological vision system capable of rapidly and automatically catching critical information. On the basis of the a priori frequency difference of different ground objects, we propose the denoising frequency attention network (DFANet) for building footprint extraction in VHR images. Specifically, we design the denoising frequency attention module and pyramid pooling module, which are embedded into the encoder–decoder network architecture. The denoising frequency attention module enables efficient filtering of high-frequency noises in the feature maps and enhancement of the frequency information related to buildings. In addition, the pyramid pooling module is leveraged to strengthen the adaptability and robustness of buildings at different scales. Experimental results of two commonly used real datasets demonstrate the effectiveness and superiority of the proposed method; the visualization and analysis also prove the critical role of the proposal.