Agricultural production is transitioning from traditional tools to IoT-connected automation devices. The integration of computer vision and agricultural automation is becoming closer with the rapid development of deep learning technology. Agricultural vision tasks, nevertheless, are relatively simple and less diverse compared to large-scale vision tasks, and there is a scarcity of plant data. This is also the reason why researchers need to make significant adjustments when directly applying advanced algorithms to the agricultural domain. In practice, downstream vision models should be designed to be concise and efficient. In this paper, we meticulously designed a deep convolutional network called TasselELANet for agricultural vision applications. It features a concise and efficient global architecture, including a 16-fold downsampling layer encoder and a decoder utilizing only 2 feature layers, which is greatly different from the general approaches. At its core is the efficient layer aggregation network, which utilizes the cross-stage fusion strategy to effectively optimize the gradient propagation path, thereby enhancing the learning ability and inference speed of the network. We validated the performance of TasselELANet on three challenging publicly available plant detection and counting datasets: maize tassels, wheat ears, and rice panels. The experimental tests achieved an average Accuracy of 0.899 and an average determination coefficient R2 of 0.867, remarkably outperforming other advanced computer vision methods in terms of accuracy and efficiency, and demonstrating sufficient generalizability. We firmly believe that TasselELANet can serve as a powerful and reliable vision tool for agricultural practitioners while also providing a solid foundation for future research. The code, datasets can be accessed at https://github.com/Ye-Sk/TasselELANet.git.