Accurately estimating rice yield is essential for ensuring global food security, enhancing agricultural productivity, and promoting agricultural economic growth. This study constructed a dataset of rice panicles at different growth stages and combined it with an attention mechanism and the YOLOv8 network to propose the YOLOv8s+LSKA+HorNet rice panicle detection and counting model, based on a drone remote sensing platform. Using the panicle count data collected by this model, along with the thousand-grain weight, number of grains per panicle, and actual yield data from a rice nitrogen gradient experimental field, various machine learning models were trained to ultimately propose a field-level rapid rice yield estimation model, RFYOLO. The experimental results show that the rice panicle detection and counting model can achieve an average precision (AP) of 98.0% and a detection speed of 20.3 milliseconds. The final yield estimation model achieved a prediction R2 value of 0.84. The detection and counting model significantly reduced missed and duplicate detections of rice panicles. Additionally, this study not only enhanced the model’s generalization ability and practicality through algorithmic innovation but also verified the impact of yield data range on the stability of the estimation model through the rice nitrogen gradient experiment. This is significant for early rice yield estimation and helping agricultural producers make more informed planting decisions.