With the development of deep learning technology, object detection has been widely applied in various fields. However, in cross-dataset object detection, conventional deep learning models often face performance degradation issues. This is particularly true in the agricultural field, where there is a multitude of crop types and a complex and variable environment. Existing technologies still face performance bottlenecks when dealing with diverse scenarios. To address these issues, this study proposes a lightweight, cross-dataset enhanced object detection method for the agricultural domain based on YOLOv9, named Multi-Adapt Recognition-YOLOv9 (MAR-YOLOv9). The traditional 32x downsampling Backbone network has been optimized, and a 16x downsampling Backbone network has been innovatively designed. A more streamlined and lightweight Main Neck structure has been introduced, along with innovative methods for feature extraction, up-sampling, and Concat connection. The hybrid connection strategy allows the model to flexibly utilize features from different levels. This solves the issues of increased training time and redundant weights caused by the detection neck and auxiliary branch structures in traditional YOLOv9, enabling MAR-YOLOv9 to maintain high performance while reducing the model’s computational complexity and improving detection speed, making it more suitable for real-time detection tasks. In comparative experiments on four plant datasets, MAR-YOLOv9 improved the mAP@0.5 accuracy by 39.18% compared to seven mainstream object detection algorithms, and by 1.28% compared to the YOLOv9 model. At the same time, the model size was reduced by 9.3%, and the number of model layers was decreased, reducing computational costs and storage requirements. Additionally, MAR-YOLOv9 demonstrated significant advantages in detecting complex agricultural images, providing an efficient, lightweight, and adaptable solution for object detection tasks in the agricultural field. The curated data and code can be accessed at the following link: https://github.com/YangxuWangamI/MAR-YOLOv9.