Remote sensing technology can be used to monitor changes in crop planting areas to guide agricultural production management and help achieve regional carbon neutrality. Agricultural UAV remote sensing technology is efficient, accurate, and flexible, which can quickly collect and transmit high-resolution data in real time to help precision agriculture management. It is widely used in crop monitoring, yield prediction, and irrigation management. However, the application of remote sensing technology faces challenges such as a high imbalance of land cover types, scarcity of labeled samples, and complex and changeable coverage types of long-term remote sensing images, which have brought great limitations to the monitoring of cultivated land cover changes. In order to solve the abovementioned problems, this paper proposed a multi-scale fusion network (MSFNet) model based on multi-scale input and feature fusion based on cultivated land time series images, and further combined MSFNet and Model Diagnostic Meta Learning (MAML) methods, using particle swarm optimization (PSO) to optimize the parameters of the neural network. The proposed method is applied to remote sensing of crops and tomatoes. The experimental results showed that the average accuracy, F1-score, and average IoU of the MSFNet model optimized by PSO + MAML (PSML) were 94.902%, 91.901%, and 90.557%, respectively. Compared with other schemes such as U-Net, PSPNet, and DeepLabv3+, this method has a better effect in solving the problem of complex ground objects and the scarcity of remote sensing image samples and provides technical support for the application of subsequent agricultural UAV remote sensing technology. The study found that the change in different crop planting areas was closely related to different climatic conditions and regional policies, which helps to guide the management of cultivated land use and provides technical support for the realization of regional carbon neutrality.