Fruit maturity is the main factor affecting the quality and yield of Camellia oil. At present, accurate and efficient detection of the maturity level of Camellia oleifera fruits in orchards and enabling selective picking for harvesting robots is a crucial issue. Aiming at the challenge of low detection efficiency of Camellia oleifera fruit maturity and complex object detection models that are difficult to deploy to Camellia oleifera selective harvesting robots, a modified lightweight You Only Look Once model (YOLO-LM) based on YOLOv7-tiny is proposed. Specifically, three Criss-Cross Attention (CCA) modules are introduced to the backbone network to reduce the influence of leaves and branches occlusion as well as mutual occlusion between fruits. Then, the Adaptively Spatial Feature Fusion (ASFF) module is applied as the head to solve the limitation of inconsistency across different feature scales on the feature pyramid. Furthermore, the GSConv is introduced to replace the standard convolution of the Neck network to enable the model to simultaneously preserve model accuracy and reduce model complexity. Experiment results demonstrate that the precision, recall, mAP@0.5, parameters, FLOPs, and model size of YOLO-LM reached 93.96 %, 93.32 %, 93.18 %, 10.17 million, 19.46 G, and 19.82 MB, which outperformed YOLOv3, Faster R-CNN (VGG16), Faster R-CNN (ResNet50), YOLOv4, and YOLOv5m. Compared to YOLOv3-tiny, YOLOv4-tiny, YOLOv5s, YOLOv7-tiny, and YOLOv8s, the mAP@0.5 of YOLO-LM increased by 6.24 %, 6.11 %, 2.52 %, 2.45 %, and 0.57 %, respectively. Meanwhile, the parameters and model size of YOLO-LM are slightly higher than that of YOLOv3-tiny, YOLOv4-tiny, YOLOv5s and YOLOv7-tiny, and significantly lower than that of YOLOv8s. In addition, the detection heatmaps of YOLOv7-tiny and YOLO-LM demonstrate these optimization operations contribute to the enhanced performance of maturity detection for Camellia oleifera fruits within natural orchard environments. Overall, the YOLO-LM model can be used for the maturity detection of Camellia oleifera fruits in orchards, which is expected to provide theoretical reference for orchard yield estimation, growth monitoring, planting management optimization and the development of Camellia oleifera selective harvesting robots.