Object detection in Unmanned Aerial Vehicles (UAV) optical remote sensing imagery presents a formidable challenge in computer vision due to the minuscule size of targets, which occupy fewer pixels and provide limited feature information, complicating accurate recognition and classification. Furthermore, the overlapping of dense targets exacerbates the difficulty of precise classification and localization. Meanwhile, classical detection networks often struggle to balance recognition accuracy with model complexity. Addressing these issues, this paper introduces EL-Net, an efficient and lightweight network model based on improvements to the YOLOv7-tiny architecture. First, the network structure is streamlined through a lightweight design that maintains performance while reducing complexity. Additionally, a feature perception enhancement module (FPEM) using attention mechanisms and dilated convolution significantly improves the model’s capability to extract key features from complex backgrounds. Finally, the optimized network structure is compressed by a structured pruning algorithm. EL-Net was evaluated in challenging scenarios on the VisDrone2019 dataset, where it achieved a mean Average Precision (mAP) of 38.7%, demonstrating high detection accuracy at minimal model complexity. Meanwhile, evaluation of the UA-DETRAC dataset has demonstrated the model’s remarkable generalization capacity. The outcomes suggest that EL-Net effectively balances accuracy and efficiency, making it ideal for deployment on resource-limited mobile edge devices while offering an innovative approach to object detection in UAV imagery.