Salient object detection aims at highlighting the most visually distinctive objects in the scene. Previous deep learning based works mainly focus on designing different integration strategies of multi-level features to improve the quality of prediction. However, due to the negligence of spatial structure coherence in predicted saliency maps, they fail to produce satisfactory results in complex scenarios. In this work, we present a structure-aware dual pyramid network (SA-DPNet) for salient object detection. By explicitly formulating spatial location information and spatial covariance features into the self-attention mechanism, a structure-aware spatial non-local block is proposed in SA-DPNet to learn the spatial-sensitive global context. With the proposed edge loss and adversarial loss, the edge structure context and patch-based global structure context are introduced to refine the structural coherence of the predicted results. Comprehensive experimental results on six RGB saliency benchmark datasets and three RGB-D saliency benchmark datasets demonstrate the superiority of proposed SA-DPNet over other state-of-the-art methods, both quantitatively and visually.