Data deficiency is one of the major issues that causes general computer vision algorithms to fail exhibiting ideal performance in agricultural applications. It is hard to get available data to train algorithms to identify different cotton growth states. In this paper, a few-shot learning algorithm was developed to recognize cotton growth states. Cotton data, collected from Awat County, Xinjiang, were initialized using an improved data augmentation method, Attn-CutMix, to highlight key image places (represent the most discriminating cotton features) and generate labels based on attention mechanism. Task-Adaptive Transformer module was proposed to automatically establish links between support and query images, and get task-specific instance embeddings to identify the current task. Plant-Pest dataset was applied to evaluate the generality of our method. Experimental results on both two datasets showed that our method outperformed different baselines. The highest accuracy of our method achieved 88% in 3-way 5-shot tasks on Cotton dataset and 94% in 3-way 10-shot tasks on Plant-Pest dataset.