Anthracnose is one of the most harmful fungal diseases in mango orchards. Accurate and rapid detection of anthracnose infection grade for mango leaves in complex natural environments is a prerequisite of precise variable pesticide spraying by robots in intelligent orchards. In this paper, A lightweight double branch inverted residual attention network (CBAM-DBIRNet) model was proposed by combining the inverted residual network and the attention mechanism, which was used to detect the infection grade of anthracnose for mango leaves in natural environment. The model designed two parallel residual branch structures to extract multi-scale features of mango leaves with different infection grades through different dilation coefficients. The inverted residual structure and depthwise separable convolution were used to reduce the complexity of the model. The CBAM attention module was integrated into the model to suppress the interference of complex background information. The mango leaves collected in the natural orchard were used for experiments. The results showed that the accuracy, recall and F1 of the CBAM-DBIRNet model for the classification of anthracnose infection grade were 98.42 %, 97.80 % and 97.89 %, respectively. The model size and parameter were 0.64 MB and 0.15 M, respectively. This study provides technical support for smart orchard management and robotic precision variable application operations.