Deep learning-based methods have achieved great success in target detection tasks of computer vision, but when it comes to Synthetic Aperture Radar (SAR) image ship detection, some new challenges appear because of the wide swath of images, diverse appearances of ships and lack of detail information, which make the detection inefficient and less effective. Aiming to these issues, in this paper, a lightweight feature optimizing network (LFO-Net) based on popular single shot detector (SSD) model is proposed for single polarization SAR image ship detection. Firstly, a simpler structure called lightweight single shot detector (LSSD) is designed, which can be trained from scratch and can reduce the training and testing time without accuracy cost. Secondly, a new bi-directional feature fusion module including one semantic aggregation block and one feature reuse block is proposed to improve the performance of multi-scale targets detection by enhancing the features of both low feature layers and high feature layers. Then the features are further optimized by leveraging attention mechanism, which is beneficial to catch the silent information more efficiently. A set of experiments are implemented to verify the effectiveness of the proposed method using the public SAR ship detection dataset (SSDD). The results show that the proposed method has significant advantages in both speed and accuracy, and outperforms other state-of-art methods. Additionally, a test on GF-3 satellite SAR data with multiple modes verifies the generalization performance of this model.