Abstract With the rapid development of aerospace and unmanned aerial vehicles, using neural networks for object detection in optical remote sensing images (O-RSI) has encountered heightened challenges. The optical remote sensing images have the characteristics of complex geometric scenes, dense groups of objects, and significant multi-scale variations of objects; researchers need to use more complex models to achieve higher accuracy. However, this complexity also brings challenges to the application of lightweight scenes. Therefore, to cope with the trade-off challenge between model complexity and detection accuracy, we propose a lightweight network model LRSDet in this study. The model integrates local and global information processing mechanisms and introduces a fast positive sample assignment strategy to adapt to resource-constrained embedded and mobile platforms. By constructing a lightweight feature extraction network and a lightweight path aggregation network and incorporating the ESM-Attention module, the feature extraction capability of the model in complex remote sensing scenarios is significantly improved. In addition, the application of a dynamic soft threshold strategy further optimizes the positive sample selection process and improves the detection efficiency of the model. Experimental on the O-RSI datasets DIOR, NWPU VHR-10, and RSOD, while analyzing model real-time performance on aerial video and embedded devices, outperforming other state-of-the-art methods.