A fast and lightweight detection algorithm is presented to improve the recognition rate and shorten the detection time of passion fruit pest detection. Based on the traditional YOLOv5 model, a new point-line distance loss function is proposed to reduce redundant computations and shorten detection time. Then, the attention module is added to the network for adaptive attention, which can focus on the target object in the channel and space dimensions to improve the detection and identification rates. Finally, the mixup online data augmentation algorithm is added to expand the online training set, which increases the model robustness and prevents over-fitting. The experimental results demonstrate the effectiveness of the proposed model. The results show the mean Average Precision is 96.51%, and the mean detection time is 7.7 ms, fulfilling the requirements of accuracy and real-time. Meanwhile, the proposed model keeps the lightweight characteristics of the traditional YOLOv5, which has a good application prospect in intelligent agriculture.