MMF-CNN: a multimodal fusion CNN network for winter wheat extraction incorporating active and passive time series data

Abstract

ABSTRACT Timely and accurate acquisition of winter wheat planting areas is crucial for food security. In this study, Sentinel-1 and Sentinel-2 time-series data are integrated at the feature level to enhance the accuracy of winter wheat extraction. However, existing feature-level fusion models suffer from insufficient feature extraction and lack of feature completeness, thereby overlooking the complementarity and correlation between these two modalities. A Multimodal Fusion Convolutional Neural Network (MMF-CNN) model is proposed to address the issues above. Firstly, the images of Sentinel-1 and Sentinel-2 are processed to obtain the NDVI and backscatter characteristics of the winter wheat time series life cycle. A single feature and a combination of two features are then imported into each end of the model. The model adds a feature fusion module, which can fully extract the feature information. At the same time, the original features are retained in the process of multiscale feature fusion, which avoids the loss of the original information. Finally, Support Vector Machine (SVM), Long Short-Term Memory (LSTM), and Visual Geometry Group (VGG) are selected as the comparison models for comparative experiments, and the classification results of remote sensing images are obtained. These results demonstrate that the joint utilization of SAR and optical data yields the highest classification accuracy, with an F1 score of 97.42% for winter wheat. The overall accuracy (OA) of the proposed MMF-CNN method in this study is 96.87%, representing a 1.86% improvement compared to the Conv1D-CNN model. This improvement signifies adaptive feature learning at different hierarchical levels. Comparing the accuracy with other mainstream methods, the OA improves by 1.75%–4.37%, reveals finer ground details, and demonstrates faster performance. This study can provide methodological references for crop extraction studies based on multi-source data and time series analysis.