The feature of large intra-class variance in fine-grained image classification is a challenge to the classification task. How to effectively learn the discriminant objects in the graph and find out the small discriminant regions is the key to classification. This paper proposes a weak-supervised fine-grained image classification algorithm based on multi-granularity feature fusion. The ECA module is fused with the classic network ResNet-50 to optimize the residual block to obtain a new basic network to enhance channel attention. Secondly, the local chaos module is introduced into the network to form a new image through random chaos regrouping so that the network can learn local regions with different scales of discrimination and obtain fine-grained feature expressions. The cooperative training of dual network branches makes the overall information and local information complement each other and have better expression. Experimental results on three widely used fine-grained image classification datasets verify that the proposed algorithm improves the accuracy of classification tasks and can effectively identify semantic sensitive features in images.