Convolutional neural networks (CNNs) have been widely applied in the context of ship detection in synthetic aperture radar (SAR) images, but the detection performance is still not ideal in scenarios with clutter interference. Mining frequency-domain information to suppress the sea clutter in SAR ship detection has attracted wide attention. However, existing frequency-domain ship detection methods do not process frequency-domain information adaptively, which results in the degradation of ship detection performance. To overcome this problem, this article proposes a novel deep learning network called YOLO-FA. YOLO-FA contains the proposed Frequency Attention Module (FAM), which can process frequency-domain information of SAR images adaptively. The proposed method can suppress the sea clutter in the SAR images with the help of frequency-domain information. We evaluate the proposed method YOLO-FA on two datasets, i.e., the High-Resolution SAR images dataset (HRSID) and SAR Ship Detection Dataset (SSDD). Compared with the baseline method YOLOv5 and existing commonly used methods, YOLO-FA achieves state-of-the-art detection performance on both datasets.