Feature Information Research Articles

Self-supervised multimodal change detection based on difference contrast learning for remote sensing imagery

Most existing change detection (CD) methods target homogeneous images. However, in real-world scenarios like disaster management, where CD is urgent and pre-changed and post-changed images are typical of different modalities, significant challenges arise for multimodal change detection (MCD). One challenge is that bi-temporal image pairs, sourced from distinct sensors, may cause an image domain gap. Another issue surfaces when multimodal bi-temporal image pairs require collaborative input from domain experts who are specialized among different image fields for pixel-level annotation, resulting in scarce annotated samples. To address these challenges, this paper proposes a novel self-supervised difference contrast learning framework (Self-DCF). This framework facilitates networks training without labelled samples by automatically exploiting the feature information inherent in bi-temporal imagery to supervise each other mutually. Additionally, a Unified Mapping Unit reduces the domain gap between different modal images. The efficiency and robustness of Self-DCF are validated on five popular datasets, outperforming state-of-the-art algorithms.

Gesture Recognition with Adaptive-Weight-Based Residual MultiheadCrossAttention Fusion Based on Multi-Level Feature Information

Gesture Recognition with Adaptive-Weight-Based Residual MultiheadCrossAttention Fusion Based on Multi-Level Feature Information

Combining multi-level feature extraction algorithm with residual graph convolutional neural network for partial discharge detection

Partial discharge (PD) identification is critical for the insulation diagnosis of cable conductors; however, there is still scope for enhancement in the existing adaptive extraction capabilities and feature utilization for PD signals. In this context, this paper introduces a method that integrates a one-dimensional convolutional neural network (1D-CNN) with a residual graph convolutional neural network (ResGCN) to recognize PD signals. Noise analysis is performed using various combinations of mother wavelets, and Bayesian optimization is employed to mitigate background noise. Key features of PD signals are progressively abstracted through a 1D-CNN-based multilevel automatic feature learning method, while signal timing attributes are maintained to minimize manual intervention. The graph data is constructed using the signal feature matrix and the signal timing feature similarity matrix. This is followed by the development of a ResGCN utilizing a graph attention mechanism to integrate node feature information and the topology of the PD graph data. This approach aims to fully exploit the correlation between local regions of the feature space and the temporal numerical properties of the signals. Additionally, it jointly optimizes feature extraction and model classification to facilitate adaptive diagnosis. The method is validated with extensive real experimental data obtained from medium voltage overhead power lines. It demonstrates exceptional performance and practicality, achieving an accuracy rate of 97.3% and a recognition rate of 96.1% for PD samples, thus offering reliable theoretical support for effective PD detection.

DSIFNet: Implicit Feature Network for Nasal Cavity and Vestibule Segmentation from 3D Head CT

This study is dedicated to accurately segment the nasal cavity and its intricate internal anatomy from head CT images, which is critical for understanding nasal physiology, diagnosing diseases, and planning surgeries. Nasal cavity and it’s anatomical structures such as the sinuses, and vestibule exhibit significant scale differences, with complex shapes and variable microstructures. These features require the segmentation method to have strong cross-scale feature extraction capabilities. To effectively address this challenge, we propose an image segmentation network named the Deeply Supervised Implicit Feature Network (DSIFNet). This network uniquely incorporates an Implicit Feature Function Module Guided by Local and Global Positional Information (LGPI-IFF), enabling effective fusion of features across scales and enhancing the network's ability to recognize details and overall structures. Additionally, we introduce a deep supervision mechanism based on implicit feature functions in the network's decoding phase, optimizing the utilization of multi-scale feature information, thus improving segmentation precision and detail representation. Furthermore, we constructed a dataset comprising 7116 CT volumes (including 1,292,508 slices) and implemented PixPro-based self-supervised pretraining to utilize unlabeled data for enhanced feature extraction. Our tests on nasal cavity and vestibule segmentation, conducted on a dataset comprising 128 head CT volumes (including 34006 slices), demonstrate the robustness and superior performance of proposed method, achieving leading results across multiple segmentation metrics.

Enhancing physician support in pancreatic cancer diagnosis: New M-F-RCNN artificial intelligence model using endoscopic ultrasound.

Background and study aims Endoscopic ultrasound (EUS) is vital for early pancreatic cancer diagnosis. Advances in artificial intelligence (AI), especially deep learning, have improved medical image analysis. We developed and validated the Modified Faster R-CNN (M-F-RCNN), an AI algorithm using EUS images to assist in diagnosing pancreatic cancer. Methods We collected EUS images from 155 patients across three endoscopy centers from July 2022 to July 2023. M-F-RCNN development involved enhancing feature information through data preprocessing and utilizing an improved Faster R-CNN model to identify cancerous regions. Its diagnostic capabilities were validated against an external set of 1,000 EUS images. In addition, five EUS doctors participated in a study comparing the M-F-RCNN model's performance with that of human experts, assessing diagnostic skill improvements with AI assistance. Results Internally, the M-F-RCNN model surpassed traditional algorithms with an average precision of 97.35%, accuracy of 96.49%, and recall rate of 5.44%. In external validation, its sensitivity, specificity, and accuracy were 91.7%, 91.5%, and 91.6%, respectively, outperforming non-expert physicians. The model also significantly enhanced the diagnostic skills of doctors. Conclusions: The M-F-RCNN model shows exceptional performance in diagnosing pancreatic cancer via EUS images, greatly improving diagnostic accuracy and efficiency, thus enhancing physician proficiency and reducing diagnostic errors.

A hybrid CNN-transformer network: Accurate and efficient semantic segmentation of crops and weeds on resource-constrained embedded devices

Weed control plays a crucial role in agricultural production. The utilization of advanced vision algorithms on intelligent weeding robots enables the autonomous and efficient resolution of weed-related issues. Vision transformers are highly sensitive to plant texture and shape, but their computational cost is too high. Consequently, we propose a novel hybrid CNN-transformer network for the semantic segmentation of crops and weeds on Resource-Constrained Embedded Devices. Our network follows an encoder–decoder structure, incorporating the proposed concat extended downsampling block in the encoder, which increases inference speed by reducing memory access time and improves the accuracy of feature extraction. For global semantic extraction, we introduce the proposed Parallel input transformer semantic enhancement module, which employs a shared transformer block to increase the computation rate. Additionally, Global-Local semantic fusion block mitigates the semantic gap problem well. To fully utilize the transformer’s ability to process plant texture and shape, we employ the fusion enhancement block in the decoder, thus minimizing the loss of feature information. Segmentation results on three publicly benchmark datasets show that our network outperforms the commonly used CNN-based, transformer-based, and hybrid CNN-transformer-based methods in terms of segmentation accuracy. Moreover, our network comprises only 0.1887M parameters and 0.2145G floating-point operations. We also evaluate the inference speed on an NVIDIA Jetson Orin NX embedded system, which result for inference single image 28.28 msec, and achieving a detection speed of 35.36 FPS. The experimental results highlight that our network maintains the best inference speed and exhibits the strongest segmentation performance on resource-constrained embedded systems.

TDOcc: Exploit machine learning and big data in multi-view 3D occupancy prediction

With the advancement of machine learning and big data technologies, BEV(Bird’s Eye View)-based methodologies have recently achieved significant breakthroughs in multi-view 3D occupancy prediction tasks. However, BEV(Bird’s Eye View)-centric 3D occupancy prediction continues to grapple with feature representation and annotation costs when applied to complex open environments. In order to surmount these issues and further propel the evolution of 3D occupancy tasks, this study introduces a novel framework termed TDOcc. By leveraging multi-camera imagery, TDOcc executes 3D semantic occupancy prediction by directly learning from unprocessed 3D spaces, thereby maximizing information retention. TDOcc presents two notable advantages: firstly, it utilizes dense occupancy labels, which not only facilitate robust dense occupancy inference but also enable comprehensive object estimation within the scene. Secondly, the framework synthesizes historical feature information by adeptly aligning past and present features through temporal cues, thereby bolstering the efficacy of the feature fusion module. Additionally, with a view to address the ill-posed nature inherent in camera-based 3D occupancy prediction, we incorporate an enhancement module that operates within the 3D feature space. This module has been meticulously crafted for the training phase to amplify the model’s learning potential. Extensive experiments conducted on the widely recognized nuScenes dataset underscore the efficacy of our proposed approach. Compared to the most recent TPVFormer and OccFormer, our approach has achieved a significant improvement in mean Intersection over Union (mIoU) by 2.0 and 0.8 respectively, and has reached performance comparable to the state-of-the-art LiDAR-based methods.

A Strip Steel Surface Defect Salient Object Detection Based on Channel, Spatial and Self-Attention Mechanisms

Strip steel is extensively utilized in industries such as automotive manufacturing and aerospace due to its superior machinability, economic benefits, and adaptability. However, defects on the surface of steel strips, such as inclusions, patches, and scratches, significantly affect the performance and service life of the product. Therefore, the salient object detection of surface defects on strip steel is crucial to ensure the quality of the final product. Many factors, such as the low contrast of surface defects on strip steel, the diversity of defect types, complex texture structures, and irregular defect distribution, hinder existing detection technologies from accurately identifying and segmenting defect areas against complex backgrounds. To address the above problems, we propose a novel detector called S3D-SOD for the salient object detection of strip steel surface defects. For the encoding stage, a residual self-attention block is proposed to explore semantic information cues of high-level features to locate and guide low-level feature information. In addition, we apply a general residual channel and spatial attention to low-level features, enabling the model to adaptively focus on the key channels and spatial areas of feature maps with high resolutions, thereby enhancing the encoder features and accelerating the convergence of the model. For the decoding stage, a simple residual decoder block with an upsampling operation is proposed to realize the integration and interaction of feature information between different layers. Here, the simple residual decoder block is used for feature integration due to the following observation: backbone networks like ResNet and the Swin Transformer, after being pretrained on the large dataset ImageNet and then fine-tuned on a smaller dataset for strip steel surface defects, are capable of extracting feature maps that contain both general image features and the specific characteristics required for the salient object detection of strip steel surface defects. The experimental results on the SD-saliency-900 dataset show that S3D-SOD is better than advanced methods, and it has strong generalization ability and robustness.

Automatic Apple Detection and Counting with AD-YOLO and MR-SORT.

In the production management of agriculture, accurate fruit counting plays a vital role in the orchard yield estimation and appropriate production decisions. Although recent tracking-by-detection algorithms have emerged as a promising fruit-counting method, they still cannot completely avoid fruit occlusion and light variations in complex orchard environments, and it is difficult to realize automatic and accurate apple counting. In this paper, a video-based multiple-object tracking method, MR-SORT (Multiple Rematching SORT), is proposed based on the improved YOLOv8 and BoT-SORT. First, we propose the AD-YOLO model, which aims to reduce the number of incorrect detections during object tracking. In the YOLOv8s backbone network, an Omni-dimensional Dynamic Convolution (ODConv) module is used to extract local feature information and enhance the model's ability better; a Global Attention Mechanism (GAM) is introduced to improve the detection ability of a foreground object (apple) in the whole image; a Soft Spatial Pyramid Pooling Layer (SSPPL) is designed to reduce the feature information dispersion and increase the sensory field of the network. Then, the improved BoT-SORT algorithm is proposed by fusing the verification mechanism, SURF feature descriptors, and the Vector of Local Aggregate Descriptors (VLAD) algorithm, which can match apples more accurately in adjacent video frames and reduce the probability of ID switching in the tracking process. The results show that the mAP metrics of the proposed AD-YOLO model are 3.1% higher than those of the YOLOv8 model, reaching 96.4%. The improved tracking algorithm has 297 fewer ID switches, which is 35.6% less than the original algorithm. The multiple-object tracking accuracy of the improved algorithm reached 85.6%, and the average counting error was reduced to 0.07. The coefficient of determination R2 between the ground truth and the predicted value reached 0.98. The above metrics show that our method can give more accurate counting results for apples and even other types of fruit.

Dual-path aggregation transformer network for super-resolution with images occlusions and variability

While Transformer-based approaches have recently achieved notable success in super-resolution, their extensive computational requirements impede widespread practical adoption. High-resolution meteorological satellite cloud imagery is essential for weather analysis and forecasting. Enhancing image resolution through super-resolution techniques facilitates the accurate identification and localization of geographic features by meteorological systems. However, current super-resolution methods fail to restore the intricacies of cloud formations and complex regions fully. This research introduces a novel dual-path aggregation Transformer network (DPAT) tailored to enhance the super-resolution of meteorological satellite cloud images. The DPAT network adeptly captures cloud imagery's subtle details and textures, effectively addressing occlusions and the variability inherent in satellite imagery. It bolsters the model's ability to manage the complex attributes of cloud images through the introduction of the Dual-path Aggregation Self-Attention (DASA) mechanism and the Multi-scale Feature Aggregation Block (MFAB), thereby enhancing performance in processing intricate cloud features. The DASA mechanism synthesizes features across spatial, depth, and channel dimensions via a dual-path approach, thoroughly exploiting feature correlations. The MFAB, designed to supplant the multilayer perceptron, incorporates shift convolution and a multi-scale interaction block to augment feature information, compensating for the deficiency in local information absorption due to fixed receptive fields. Experimental outcomes indicate that DPAT delivers superior super-resolution outcomes. With a parameter count of only 32% of the Enhanced Deep Residual Network (EDSR) or 77% of the Image Restoration using Shift Window Transformer (SwinIR), DPAT matches SwinIR's performance on the satellite cloud dataset. Moreover, DPAT balances accuracy and parameter economy across various datasets. This technology is expected to improve image super-resolution capabilities in multiple fields such as human action recognition and industrial recognition, and indirectly improve the accuracy of image perception tasks.

Hierarchical-Concatenate Fusion TDNN for sound event classification.

Semantic feature combination/parsing issue is one of the key problems in sound event classification for acoustic scene analysis, environmental sound monitoring, and urban soundscape analysis. The input audio signal in the acoustic scene classification is composed of multiple acoustic events, which usually leads to low recognition rate in complex environments. To address this issue, this paper proposes the Hierarchical-Concatenate Fusion(HCF)-TDNN model by adding HCF Module to ECAPA-TDNN model for sound event classification. In the HCF module, firstly, the audio signal is converted into two-dimensional time-frequency features for segmentation. Then, the segmented features are convolved one by one for improving the small receptive field in perceiving details. Finally, after the convolution is completed, the two adjacent parts are combined before proceeding with the next convolution for enlarging the receptive field in capturing large targets. Therefore, the improved model further enhances the scalability by emphasizing channel attention and efficient propagation and aggregation of feature information. The proposed model is trained and validated on the Urbansound8K dataset. The experimental results show that the proposed model can achieve the best classification accuracy of 95.83%, which is an approximate improvement of 5% (relatively) over the ECAPA-TDNN model.

Segmentation-Based Detection for Luffa Seedling Grading Using the Seg-FL Model

This study addresses the issue of inaccurate and error-prone grading judgments in luffa plug seedlings. A new Seg-FL seedling segmentation model is proposed as an extension of the YOLOv5s-Seg model. The small leaves of early-stage luffa seedlings are liable to be mistaken for impurities in the plug trays. To address this issue, cross-scale connections and weighted feature fusion are introduced in order to integrate feature information from different levels, thereby improving the recognition and segmentation accuracy of seedlings or details by refining the PANet structure. To address the ambiguity of seedling edge information during segmentation, an efficient channel attention module is incorporated to enhance the network’s focus on seedling edge information and suppress irrelevant features, thus sharpening the model’s focus on luffa seedlings. By optimizing the CIoU loss function, the calculation of overlapping areas, center point distances, and aspect ratios between predicted and ground truth boxes is preserved, thereby accelerating the convergence process and reducing the computational resource requirements on edge devices. The experimental results demonstrate that the proposed model attains a mean average precision of 97.03% on a self-compiled luffa plug seedling dataset, representing a 6.23 percentage point improvement over the original YOLOv5s-Seg. Furthermore, compared to the YOLACT++, FCN, and Mask R-CNN segmentation models, the improved model displays increases in mAP@0.5 of 12.93%, 13.73%, and 10.53%, respectively, and improvements in precision of 15.73%, 16.93%, and 13.33%, respectively. This research not only validates the viability of the enhanced model for luffa seedling grading but also provides tangible technical support for the automation of grading in agricultural production.

A graph convolutional neural network model based on fused multi-subgraph as input and fused feature information as output

The graph convolution neural network (GCN)-based node classification model tackles the challenge of classifying nodes in graph data through learned feature representations. However, most existing graph neural networks primarily focus on the same type of edges, which might not accurately reflect the intricate real-world graph structure. This paper introduces a novel graph neural network model, MF-GCN, which integrates subgraphs with various edge types as input and combines feature information from each graph convolutional neural network layer to produce the final output. This model learns node feature representations by separately feeding subgraphs with different edge types into the graph convolutional layer. It then computes the weight vectors for fusing various edge type subgraphs based on the learned node features. Additionally, to efficiently extract feature information, the outputs of each graph convolution layer, without an activation function, are weighted and summed to obtain the final node features. This approach resolves the challenges of determining fusion weights and effectively extracting feature information during subgraph fusion. Experimental results show that the proposed model significantly improves performance on all three datasets, highlighting its effectiveness in node representation learning tasks.

TLCSFI: A Pose-Guided Person Re-Identification Method with Two-Level Channel–Spatial Feature Integration

Person re-identification methods currently encounter challenges in feature learning, primarily due to difficulties in expressing the correlation between local features and integrating global and local features effectively. To address these issues, a pose-guided person re-identification method with Two-Level Channel–Spatial Feature Integration (TLCSFI) is proposed. In TLCSFI, a two-level integration mechanism is implemented. At the first level, TLCSFI integrates the spatial information from local features to generate fine-grained spatial features. At the second level, the fine-grained spatial feature and the coarse-grained channel feature are integrated together to complete channel–spatial feature integration. In the method, a Pose-based Spatial Feature Integration (PSFI) module is introduced to generate the pose union feature, which calculates intra-body affinity to guide the integration of spatial information among local pose feature maps. Then, a Channel and Spatial Union Feature Integration (CSUFI) module is proposed to efficiently integrate the channel information of the global feature and the spatial information of the pose union feature. Two individual networks are designed to extract channel and spatial information, respectively, in CSUFI, which are then weighted and integrated. Experiments are conducted on three publicly available datasets to evaluate TLCSFI, and the experimental results demonstrate its competitive performance.

Lightweight and efficient deep learning models for fruit detection in orchards

The accurate recognition of apples in complex orchard environments is a fundamental aspect of the operation of automated picking equipment. This paper aims to investigate the influence of dense targets, occlusion, and the natural environment in practical application scenarios. To this end, it constructs a fruit dataset containing different scenarios and proposes a real-time lightweight detection network, ELD(Efficient Lightweight object Detector). The EGSS(Efficient Ghost-shuffle Slim module) module and MCAttention(Mix channel Attention) are proposed as innovative solutions to the problems of feature extraction and classification. The attention mechanism is employed to construct a novel feature extraction network, which effectively utilizes the low-latitude feature information, significantly enhances the fine-grained feature information and gradient flow of the model, and improves the model’s anti-interference ability. Eliminate redundant channels with SlimPAN to further compress the network and optimise functionality. The network as a whole employs the Shape-IOU loss function, which considers the influence of the bounding box itself, thereby enhancing the robustness of the model. Finally, the target detection accuracy is enhanced through the transfer of knowledge from the teacher’s network through knowledge distillation, while ensuring that the overall network is sufficiently lightweight. The experimental results demonstrate that the ELD network, designed for fruit detection, achieves an accuracy of 87.4%. It has a relatively low number of parameters (4.3×105\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$4.3 \ imes 10^5$$\\end{document}), a GLOPs of only 1.7, and a high FPS of 156. This network can achieve high accuracy while consuming fewer computational resources and performing better than other networks.

MS-YOLO: A Lightweight and High-Precision YOLO Model for Drowning Detection.

A novel detection model, MS-YOLO, is developed in this paper to improve the efficiency of drowning rescue operations. The model is lightweight, high in precision, and applicable for intelligent hardware platforms. Firstly, the MD-C2F structure is built to capture the subtle movements and posture changes in various aquatic environments, with a light weight achieved by introducing dynamic convolution (DcConv). To make the model perform better in small object detection, the EMA mechanism is incorporated into the MD-C2F. Secondly, the MSI-SPPF module is constructed to improve the performance in identifying the features of different scales and the understanding of complex backgrounds. Finally, the ConCat single-channel fusion is replaced by BiFPN weighted channel fusion to retain more feature information and remove the irrelevant information in drowning features. Relative to the Faster R-CNN, SSD, YOLOv6, YOLOv9, and YOLOv10, the MS-YOLO achieves an average accuracy of 86.4% in detection on a self-built dataset at an ultra-low computational cost of 7.3 GFLOPs.

An improved YOLOv8n-IRP model for natural rubber tree tapping surface detection and tapping key point positioning.

Aiming at the problem that lightweight algorithm models are difficult to accurately detect and locate tapping surfaces and tapping key points in complex rubber forest environments, this paper proposes an improved YOLOv8n-IRP model based on the YOLOv8n-Pose. First, the receptive field attention mechanism is introduced into the backbone network to enhance the feature extraction ability of the tapping surface. Secondly, the AFPN structure is used to reduce the loss and degradation of the low-level and high-level feature information. Finally, this paper designs a dual-branch key point detection head to improve the screening ability of key point features in the tapping surface. In the detection performance comparison experiment, the YOLOv8n-IRP improves the D_mAP50 and P_mAP50 by 1.4% and 2.3%, respectively, over the original model while achieving an average detection success rate of 87% in the variable illumination test, which demonstrates enhanced robustness. In the positioning performance comparison experiment, the YOLOv8n-IRP achieves an overall better localization performance than YOLOv8n-Pose and YOLOv5n-Pose, realizing an average Euclidean distance error of less than 40 pixels. In summary, YOLOv8n-IRP shows excellent detection and positioning performance, which not only provides a new method for the key point localization of the rubber-tapping robot but also provides technical support for the unmanned rubber-tapping operation of the intelligent rubber-tapping robot.

Multi-Dimensional Fuzzy Clustering-Based Trajectory Initialization Algorithm for Infrared Weak Target Trajectories in Robust Clutter Environments

When conducting maneuver target tracking, trajectory initialization plays a crucial role in enhancing the accuracy of tracking algorithms. During maneuver target tracking, the accuracy of the tracking algorithm can be significantly improved through trajectory initialization. However, the traditional trajectory initialization algorithms face issues such as susceptibility to noise interference, lack of universality, and poor robustness in environments with high clutter levels. To address these issues, this study proposes a trajectory initialization algorithm based on multidimensional fuzzy clustering (MDF-clustering). The algorithm utilizes multidimensional feature information of the target, such as speed and irradiance, to determine point trajectory affiliation by assigning weights based on the clustering center of each feature type. Subsequently, it updates the clustering center and weight assignment using the new target features, ultimately deriving the correct trajectory through iterative processes. Experimental results demonstrate that the proposed method achieves an average stable initialization frame number of 3.12 frames, an average correct trajectory initialization rate of 99.59%, an average false trajectory occupancy rate of 0.04%, and an average missed batch rate of 0.06%. These results indicate improvements of at least 0.87 frames, 27.11%, 60.28%, and 6.48%, respectively, in terms of initialization rate, false trajectory rate, and missed batch rate, when compared to traditional methods. The algorithm enhances the accuracy and robustness of trajectory initialization in challenging environments characterized by solid clutter and target maneuvers, offering significant practical value for target tracking in complex scenarios.

A Novel RUL-Centric Data Augmentation Method for Predicting the Remaining Useful Life of Bearings

Maintaining the reliability of rotating machinery in industrial environments entails significant challenges. The objective of this paper is to develop a methodology that can accurately predict the condition of rotating machinery in order to facilitate the implementation of effective preventive maintenance strategies. This article proposed a novel RUL-centric data augmentation method, designated as DF-MDAGRU, for the purpose of predicting the remaining useful life (RUL) of bearings. This model is based on an encoder–decoder framework that integrates time–frequency domain feature enhancement with multidimensional dynamic attention gated recurrent units for feature extraction. This method enhances time–frequency domain features through the Discrete Wavelet Downsampling module (DWD) and Convolutional Fourier Residual Block (CFRB). This method employs a Multiscale Channel Attention Module (MS-CAM) and a Multiscale Convolutional Spatial Attention Mechanism (MSSAM) to extract channel and spatial feature information. Finally, the output predictions are processed through linear regression to achieve the final RUL estimation. Experimental results demonstrate that the proposed method outperforms other state-of-the-art approaches on the FEMETO-ST and XJTU datasets.

Small-sample cucumber disease identification based on multimodal self-supervised learning

It is difficult and costly to obtain large-scale, labeled crop disease data in the field of agriculture. How to use small samples of unlabeled data for feature learning has become an urgent problem that needs to be solved. The emergence of self-supervised contrastive learning methods and self-supervised mask learning methods can solve the problem of missing labels on the training data. However, each of these paradigms comes with its own advantages and drawbacks. At the same time, the features learned by dataset in a single modality are limited, ignoring the correlation with other modal information. Hence, this paper introduced an effective framework for multimodal self-supervised learning, denoted as MMSSL, to address the task of identifying cucumber diseases with small sample sizes. Integrating image self-supervised mask learning, image self-supervised contrastive learning, and multimodal image-text contrastive learning, the model can not only learn disease feature information from different modalities, but also capture global and local disease feature information. Simultaneously, the mask learning branch was enhanced by introducing a prompt learning module based on a cross-attention network. This module aided in approximately locating the masked regions in the image data in advance, facilitating the decoder in making accurate decoding predictions. Experimental results demonstrate that the proposed method achieves a 95% accuracy in cucumber disease identification in the absence of labels. The approach effectively uncovers high-level semantic features within multimodal small-sample cucumber disease data. GradCAM is also employed for visual analysis to further understand the decision-making process of the model in disease identification. In conclusion, the proposed method in this paper is advantageous for enhancing the classification accuracy of small-sample cucumber data in a multimodal, unlabeled context, demonstrating good generalization performance.