High Inter-class Similarity Research Articles

NmODE-Unet: A Novel Network for Semantic Segmentation of Medical Images

Diabetic retinopathy is a prevalent eye disease that poses a potential risk of blindness. Nevertheless, due to the small size of diabetic retinopathy lesions and the high interclass similarity in terms of location, color, and shape among different lesions, the segmentation task is highly challenging. To address these issues, we proposed a novel framework named nmODE-Unet, which is based on the nmODE (neural memory Ordinary Differential Equation) block and U-net backbone. In nmODE-Unet, the shallow features serve as input to the nmODE block, and the output of the nmODE block is fused with the corresponding deep features. Extensive experiments were conducted on the IDRiD dataset, e_ophtha dataset, and the LGG segmentation dataset, and the results demonstrate that, in comparison to other competing models, nmODE-Unet showcases a superior performance.

Deep Augmented Metric Learning Network for Prostate Cancer Classification in Ultrasound Images.

Prostate cancer screening often relies on cost-intensive MRIs and invasive needle biopsies. Transrectal ultrasound imaging, as a more affordable and non-invasive alternative, faces the challenge of high inter-class similarity and intra-class variability between benign and malignant prostate cancers. This complexity requires more stringent differentiation of subtle features for accurate auxiliary diagnosis. In response, we introduce the novel Deep Augmented Metric Learning (DAML) network, specifically tailored for ultrasound-based prostate cancer classification. The DAML network represents a significant innovation in the metric learning space, introducing the Semantic Differences Mining Strategy (SDMS) to effectively discern and represent subtle differences in prostate ultrasound images, thereby enhancing tumor classification accuracy. Additionally, the DAML network strategically addresses class variability and limited sample sizes by combining the Linear Interpolation Augmentation Strategy (LIAS) and Permutation-Aided Reconstruction Loss (PARL). This approach enriches feature representation and introduces variability with straightforward structures, mirroring the efficacy of advanced sample generation techniques. We carried out comprehensive empirical assessments of the DAML model by testing its key components against a range of models, ensuring its effectiveness. Our results demonstrate the enhanced performance of the DAML model, achieving classification accuracies of 0.857 and 0.888 for benign and malignant cancers, respectively, underscoring its effectiveness in prostate cancer classification via medical imaging.

Learn by Yourself: A Feature-Augmented Self-Distillation Convolutional Neural Network for Remote Sensing Scene Image Classification

In recent years, with the rapid development of deep learning technology, great progress has been made in remote sensing scene image classification. Compared with natural images, remote sensing scene images are usually more complex, with high inter-class similarity and large intra-class differences, which makes it difficult for commonly used networks to effectively learn the features of remote sensing scene images. In addition, most existing methods adopt hard labels to supervise the network model, which makes the model prone to losing fine-grained information of ground objects. In order to solve these problems, a feature-augmented self-distilled convolutional neural network (FASDNet) is proposed. First, ResNet34 is adopted as the backbone network to extract multi-level features of images. Next, a feature augmentation pyramid module (FAPM) is designed to extract and fuse multi-level feature information. Then, auxiliary branches are constructed to provide additional supervision information. The self-distillation method is utilized between the feature augmentation pyramid module and the backbone network, as well as between the backbone network and auxiliary branches. Finally, the proposed model is jointly supervised using feature distillation loss, logits distillation loss, and cross-entropy loss. A lot of experiments are conducted on four widely used remote sensing scene image datasets, and the experimental results show that the proposed method is superior to some state-ot-the-art classification methods.

Cross-Level Multi-Modal Features Learning With Transformer for RGB-D Object Recognition

Object recognition, one of the main goals of robot vision, is a vital prerequisite for service robots to perform domestic tasks. Thanks to the rich sense of information provided by RGB-D sensors, RGB-D-based object recognition has received increasing attention. However, the existing works focus on collaborative RGB and depth data for object recognition, while ignoring the influence of depth image quality on recognition performance. Moreover, in real-world scenarios, there are many objects with strong similarity from certain observation angles, which poses a challenge for the service robot to recognize objects accurately. In this paper, we propose CNN-TransNet, a novel end-to-end Transformer-based architecture with convolutional neural networks (CNNs) for RGB-D object recognition. In order to deal with the effect of high inter-class similarity, discriminative multi-modal feature representations are generated by learning and relating multi-modal features at multiple levels. Besides, we employ a multi-modal fusion and projection (MMFP) module to reweight the contribution of each modality to address the problem of poor-quality depth image. Our proposed approach achieves state-of-the-art performance on three datasets (including Washington RGB-D Object Dataset, JHUIT-50, and Object Clutter Indoor Dataset), with accuracy of 95.4%, 98.1%, and 94.7%, respectively. The results demonstrate the effectiveness and superiority of the proposed model in RGB-D object recognition task.

PCA: Progressive class-wise attention for skin lesions diagnosis

Skin cancer is the most prevalent type of cancer worldwide. Early detection is essential as it could be fatal at later stages. The classification of skin lesions is challenging since there are many variations, including changes in color, shape, size, high intra-class variation, and high inter-class similarity. In this paper, a unique class-wise attention method is proposed that considers each class equally while extracting additional discriminative information of skin lesions. The proposed attention mechanism is employed in a progressive manner to incorporate discriminative feature information from multiple scales. The proposed approach obtained competitive performance against more than 15 state-of-the-art methods including HAM1000 and ISIC 2019 leaderboard winners. The proposed method achieved 97.40% accuracy on the HAM10000 and 94.9% accuracy on the ISIC 2019 dataset.

Lithological mapping of geological remote sensing via adversarial semi-supervised segmentation network

Geological remote sensing interpretation (GRSI), which aims to recognize multiple geological elements based on their characteristics on satellite remote sensing images, is vital in large-scale regional lithological mapping. However, due to the influence of long-term geological movements, the spatial distribution of geological elements (such as lithology, glaciers, and soils) on the image is often complex and highly fragmented. In addition, the characteristics of high inter-class similarity and severe homogenization make the annotation of geological element samples require significant cost and expert knowledge. These lead to insufficient interpretation accuracy and limited annotation samples in GRSI. To alleviate the dependence of labeled samples and promote the performance of GRSI, we propose an adversarial semi-supervised segmentation network with object-context and global-attention (AdvSemi-OCGNet) for the GRSI, which achieves effective segmentation results in the case of limited labeled samples. Under the architecture of adversarial learning, a proposed baseline network OCGNet and a full convolution discriminator (FCD) are integrated to conduct semi-supervised segmentation. OCGNet, as the generator, aims to confuse FCD by predicting probability maps. Then FCD selects trustworthy regions with high-confidence of unlabeled sample to generate pseudo-labels for semi-supervised segmentation of OCGNet. Iterative adversarial learning is employed to simulate the process of expert interpretation of geological elements through continuous discrimination and correction. Finally, a fully connected conditional random field eliminates holes and isolated areas of segmented results caused by misclassification. Two study areas are selected, which include various geological elements such as multiple lithologies, soils, rock glaciers, and surface water. Numerous experiments have revealed the superiority of the proposed model in GRSI with limited annotation samples.

ARM-Net: Attention-guided residual multiscale CNN for multiclass brain tumor classification using MR images

Brain tumor is the deadliest type of cancer and has the lowest survival rate when compared with other cancers. Hence, timely detection of brain tumor is indispensable for patients to make better treatment plans, leading to improved life expectancy. However, accurate classification of different brain tumor types from MR images is challenging due to high inter-class similarities. Though deep learning architectures, mainly CNNs, have shown promising performance compared to traditional approaches, such models often demand huge parameters and lead to overfitting while dealing with limited training samples. Further, the state-of-the-art CNN models cannot capture the subtle lesion size and shape variations among different classes. To cope with these issues, in this paper, we propose an attention-based residual multiscale CNN called ARM-Net for multiclass brain tumor classification. In particular, we propose a lightweight residual multiscale CNN dubbed RM-Net to capture high-level feature representations at different receptive fields. Further, a lightweight global attention module (LGAM) is proposed to selectively learn more discriminative features. The LGAM is placed on the top of RM-Net and is introduced to capture wide-range feature dependencies. Experimental results on two benchmark datasets indicate the superiority of our ARM-Net over the state-of-the-art CNN architectures and existing methods. The ARM-Net achieves an accuracy of 96.64% and 97.11% on MBTD and BraTS 2020 dataset, respectively. The ablation studies, Grad-CAM, and Grad-CAM++ visualization results confirm the effectiveness of our proposed LGAM. In addition, our ARM-Net is lightweight, end-to-end learnable, and hence more suitable for real-time brain tumor classification.

Adaptive Adjacent Layer Feature Fusion for Object Detection in Remote Sensing Images

Object detection in remote sensing images faces the challenges of a complex background, large object size variations, and high inter-class similarity. To address these problems, we propose an adaptive adjacent layer feature fusion (AALFF) method, which is developed on the basis of RTMDet. Specifically, the AALFF method incorporates an adjacent layer feature fusion enhancement (ALFFE) module, designed to capture high-level semantic information and accurately locate object spatial positions. ALFFE also effectively preserves small objects by fusing adjacent layer features and employs involution to aggregate contextual information in a wide spatial range for object essential features extraction in complex backgrounds. Additionally, the adaptive spatial feature fusion (ASFF) module is introduced to guide the network to select and fuse the crucial features to improve the adaptability to objects with different sizes. The proposed method achieves mean average precision (mAP) values of 77.1%, 88.9%, and 95.7% on the DIOR, HRRSD, and NWPU VHR-10 datasets, respectively. Notably, our approach achieves mAP75 values of 60.8% and 79.0% on the DIOR and HRRSD datasets, respectively, surpassing the state-of-the-art performance on the DIOR dataset.

VGG16-based intelligent image analysis in the pathological diagnosis of IgA nephropathy

ObjectiveIgA nephropathy (IgAN) is the most common primary glomerular disease worldwide, with heterogeneous clinical and pathological manifestations, and is a common cause of end-stage renal disease. Early detection and effective intervention measures are essential for improving the outcome of IgAN. Machine learning methods can make the pathological analysis, early detection and diagnosis, and prognosis prediction of IgAN more automated and accurate. This article discusses the application of machine learning methods in IgAN, from optimizing pathological diagnosis to discovering non-invasive specific biomarkers, predicting disease progression, and evaluating prognosis. It is the key to reducing the incidence rate and mortality of end-stage renal disease by relying on intelligent image analysis of VGG16 for accurate detection and diagnosis of IgA nephropathy to enable clinical to take effective prevention and treatment measures. MethodsA total of 452 cases of kidney disease admitted to the nephrology department of our hospital from January 2018 to February 2023 were selected, and it was ruled out that pathological diagnosis could not be made due to the small number of samples submitted for renal puncture; After excluding suspected cases of renal biopsy pathology diagnosis and patients who did not undergo immunofluorescence examination, a total of 135 confirmed IgA nephropathy patients were subjected to image analysis. The internationally recognized 5-level semi-quantitative method was used for evaluation, and traditional image processing methods were selected to segment and extract fluorescence deposition areas. Transform the input image into color space and generate a binary image using the adaptive threshold method in the two feature dimensions of color and brightness. Then, VGG16 regions were separated and merged to obtain independent sedimentary regions. VGG16 was used to add BN layers and SE visual attention to fully extract sensitive features with high inter-class similarity and significant intra-class differences in the IgA nephropathy image classification task. The contour, area, and average brightness of each sedimentary region were calculated, and automatic computer recognition of fluorescence deposition intensity and shape was obtained to improve the accuracy of IgA nephropathy image classification. ResultsThe artificial intelligence image analysis based on VGG16 can achieve the interpretation of IgA nephropathy immunofluorescence results with a higher coincidence rate compared to the results of pathological diagnostic doctors. IgA reaches 88.9%, IgG reaches 85.8%, IgM reaches 83.8%, and C3 reaches 88.6%. Therefore, it can assist pathological diagnostic doctors in interpreting IgA nephropathy immunofluorescence. ConclusionBy fully utilizing computer and network technologies to change the pathological workflow, improve the work efficiency of pathological diagnostic doctors, reduce the misdiagnosis rate caused by fatigue during film reading, and make pathological diagnosis more accurate and objective.

DB-DCAFN: dual-branch deformable cross-attention fusion network for bacterial segmentation

Sputum smear tests are critical for the diagnosis of respiratory diseases. Automatic segmentation of bacteria from sputum smear images is important for improving diagnostic efficiency. However, this remains a challenging task owing to the high interclass similarity among different categories of bacteria and the low contrast of the bacterial edges. To explore more levels of global pattern features to promote the distinguishing ability of bacterial categories and maintain sufficient local fine-grained features to ensure accurate localization of ambiguous bacteria simultaneously, we propose a novel dual-branch deformable cross-attention fusion network (DB-DCAFN) for accurate bacterial segmentation. Specifically, we first designed a dual-branch encoder consisting of multiple convolution and transformer blocks in parallel to simultaneously extract multilevel local and global features. We then designed a sparse and deformable cross-attention module to capture the semantic dependencies between local and global features, which can bridge the semantic gap and fuse features effectively. Furthermore, we designed a feature assignment fusion module to enhance meaningful features using an adaptive feature weighting strategy to obtain more accurate segmentation. We conducted extensive experiments to evaluate the effectiveness of DB-DCAFN on a clinical dataset comprising three bacterial categories: Acinetobacter baumannii, Klebsiella pneumoniae, and Pseudomonas aeruginosa. The experimental results demonstrate that the proposed DB-DCAFN outperforms other state-of-the-art methods and is effective at segmenting bacteria from sputum smear images.

State-of-the-Art Deep Learning Methods for Objects Detection in Remote Sensing Satellite Images.

Introduction: Object detection in remotely sensed satellite images is critical to socio-economic, bio-physical, and environmental monitoring, necessary for the prevention of natural disasters such as flooding and fires, socio-economic service delivery, and general urban and rural planning and management. Whereas deep learning approaches have recently gained popularity in remotely sensed image analysis, they have been unable to efficiently detect image objects due to complex landscape heterogeneity, high inter-class similarity and intra-class diversity, and difficulty in acquiring suitable training data that represents the complexities, among others. Methods: To address these challenges, this study employed multi-object detection deep learning algorithms with a transfer learning approach on remotely sensed satellite imagery captured on a heterogeneous landscape. In the study, a new dataset of diverse features with five object classes collected from Google Earth Engine in various locations in southern KwaZulu-Natal province in South Africa was used to evaluate the models. The dataset images were characterized with objects that have varying sizes and resolutions. Five (5) object detection methods based on R-CNN and YOLO architectures were investigated via experiments on our newly created dataset. Conclusions: This paper provides a comprehensive performance evaluation and analysis of the recent deep learning-based object detection methods for detecting objects in high-resolution remote sensing satellite images. The models were also evaluated on two publicly available datasets: Visdron and PASCAL VOC2007. Results showed that the highest detection accuracy of the vegetation and swimming pool instances was more than 90%, and the fastest detection speed 0.2 ms was observed in YOLOv8.

A discriminative shape-texture convolutional neural network for early diagnosis of knee osteoarthritis from X-ray images.

Knee Osteoarthritis (OA) is one of the most common causes of physical disability worldwide associated with a significant personal and socioeconomic burden. Deep Learning approaches based on Convolutional Neural Networks (CNNs) achieved remarkable improvements in knee OA detection. Despite this success, the problem of early knee OA diagnosis from plain radiographs remains a challenging task. This is due to the high similarity between the X-ray images of OA and non-OA subjects and the disappearance of texture information regarding bone microarchitecture changes in the top layers during the learning process of the CNN models. To address these issues, we propose a Discriminative Shape-Texture Convolutional Neural Network (DST-CNN), which automatically diagnoses early knee OA from X-ray images. The proposed model incorporates a discriminative loss to improve class separability and deal with high inter-class similarities. In addition, a new Gram Matrix Descriptor (GMD) block is embedded in the CNN architecture to compute texture features from several intermediate layers and combine them with the shape features in the top layers. We show that merging texture features with deep ones leads to better prediction of the early stages of OA. Comprehensive experimental results on two large public databases, Osteoarthritis Initiative (OAI) and Multicenter Osteoarthritis Study (MOST) demonstrate the potential of the proposed network. Ablation studies and visualizations are provided for a detailed understanding of our proposed approach.

OView-AI Supporter for Classifying Pneumonia, Pneumothorax, Tuberculosis, Lung Cancer Chest X-ray Images Using Multi-Stage Superpixels Classification.

The deep learning approach has recently attracted much attention for its outstanding performance to assist in clinical diagnostic tasks, notably in computer-aided solutions. Computer-aided solutions are being developed using chest radiography to identify lung diseases. A chest X-ray image is one of the most often utilized diagnostic imaging modalities in computer-aided solutions since it produces non-invasive standard-of-care data. However, the accurate identification of a specific illness in chest X-ray images still poses a challenge due to their high inter-class similarities and low intra-class variant abnormalities, especially given the complex nature of radiographs and the complex anatomy of the chest. In this paper, we proposed a deep-learning-based solution to classify four lung diseases (pneumonia, pneumothorax, tuberculosis, and lung cancer) and healthy lungs using chest X-ray images. In order to achieve a high performance, the EfficientNet B7 model with the pre-trained weights of ImageNet trained by Noisy Student was used as a backbone model, followed by our proposed fine-tuned layers and hyperparameters. Our study achieved an average test accuracy of 97.42%, sensitivity of 95.93%, and specificity of 99.05%. Additionally, our findings were utilized as diagnostic supporting software in OView-AI system (computer-aided application). We conducted 910 clinical trials and achieved an AUC confidence interval (95% CI) of the diagnostic results in the OView-AI system of 97.01%, sensitivity of 95.68%, and specificity of 99.34%.

Satellite Image Categorization Using Scalable Deep Learning

Detecting and classifying objects from satellite images are crucial for many applications, ranging from marine monitoring to land planning, ecology to warfare, etc. Spatial and temporal information-rich satellite images are exploited in a variety of manners to solve many real-world remote sensing problems. Satellite image classification has many associated challenges. These challenges include data availability, the quality of data, the quantity of data, and data distribution. These challenges make the analysis of satellite images more challenging. A convolutional neural network architecture with a scaling method is proposed for the classification of satellite images. The scaling method can evenly scale all dimensions of depth, width, and resolution using a compound coefficient. It can be used as a preliminary task in urban planning, satellite surveillance, monitoring, etc. It can also be helpful in geo-information and maritime monitoring systems. The proposed methodology is based on an end-to-end, scalable satellite image interpretation. It uses spatial information from satellite images to categorize these into four categories. The proposed method gives encouraging and promising results on a challenging dataset with a high inter-class similarity and intra-class variation. The proposed method shows 99.64% accuracy on the RSI-CB256 dataset.

P2FEViT: Plug-and-Play CNN Feature Embedded Hybrid Vision Transformer for Remote Sensing Image Classification

Remote sensing image classification (RSIC) is a classical and fundamental task in the intelligent interpretation of remote sensing imagery, which can provide unique labeling information for each acquired remote sensing image. Thanks to the potent global context information extraction ability of the multi-head self-attention (MSA) mechanism, visual transformer (ViT)-based architectures have shown excellent capability in natural scene image classification. However, in order to achieve powerful RSIC performance, it is insufficient to capture global spatial information alone. Specifically, for fine-grained target recognition tasks with high inter-class similarity, discriminative and effective local feature representations are key to correct classification. In addition, due to the lack of inductive biases, the powerful global spatial context representation capability of ViT requires lengthy training procedures and large-scale pre-training data volume. To solve the above problems, a hybrid architecture of convolution neural network (CNN) and ViT is proposed to improve the RSIC ability, called P2FEViT, which integrates plug-and-play CNN features with ViT. In this paper, the feature representation capabilities of CNN and ViT applying for RSIC are first analyzed. Second, aiming to integrate the advantages of CNN and ViT, a novel approach embedding CNN features into the ViT architecture is proposed, which can make the model synchronously capture and fuse global context and local multimodal information to further improve the classification capability of ViT. Third, based on the hybrid structure, only a simple cross-entropy loss is employed for model training. The model can also have rapid and comfortable convergence with relatively less training data than the original ViT. Finally, extensive experiments are conducted on the public and challenging remote sensing scene classification dataset of NWPU-RESISC45 (NWPU-R45) and the self-built fine-grained target classification dataset called BIT-AFGR50. The experimental results demonstrate that the proposed P2FEViT can effectively improve the feature description capability and obtain outstanding image classification performance, while significantly reducing the high dependence of ViT on large-scale pre-training data volume and accelerating the convergence speed. The code and self-built dataset will be released at our webpages.

FDTrans: Frequency Domain Transformer Model for predicting subtypes of lung cancer using multimodal data

Background and purpose:Accurate identification of lung cancer subtypes in medical images is of great significance for the diagnosis and treatment of lung cancer. Despite substantial progress in existing methods, they remain challenging due to limited annotated datasets, large intra-class differences, and high inter-class similarities. Methods:To address these challenges, we propose a Frequency Domain Transformer Model (FDTrans) to identify patients’ lung cancer subtypes using the TCGA lung cancer dataset. We add a pre-processing process to transfer histopathological images to the frequency domain using a block-based discrete cosine transform and design a coordinate Coordinate-Spatial Attention Module (CSAM) to obtain critical detail information by reassigning weights to the location information and channel information of different frequency vectors. Then, a Cross-Domain Transformer Block (CDTB) is designed for Y, Cb, and Cr channel features, capturing the long-term dependencies and global contextual connections between different component features. At the same time, feature extraction is performed on the genomic data to obtain specific features. Finally, the image branch and the gene branch are fused, and the classification result is output through the fully connected layer. Results:In 10-fold cross-validation, the method achieves an AUC of 93.16% and overall accuracy of 92.33%, which is better than similar current lung cancer subtypes classification detection methods. Conclusion:This method can help physicians diagnose the subtypes classification of lung cancer in patients and can benefit from both spatial and frequency domain information.

Merging nucleus datasets by correlation-based cross-training

Fine-grained nucleus classification is challenging because of the high inter-class similarity and intra-class variability. Therefore, a large number of labeled data is required for training effective nucleus classification models. However, it is challenging to label a large-scale nucleus classification dataset comparable to ImageNet in natural images, considering that high-quality nucleus labeling requires specific domain knowledge. In addition, the existing publicly available datasets are often inconsistently labeled with divergent labeling criteria. Due to this inconsistency, conventional models have to be trained on each dataset separately and work independently to infer their own classification results, limiting their classification performance. To fully utilize all annotated datasets, we formulate the nucleus classification task as a multi-label problem with missing labels to utilize all datasets in a unified framework. Specifically, we merge all datasets and combine their labels as multiple labels. Thus, each data has one ground-truth label and several missing labels. We devise a base classification module that is trained using all data but sparsely supervised by the ground-truth labels only. We then exploit the correlation among different label sets by a label correlation module. By doing so, we can have two trained basic modules and further cross-train them with both ground-truth labels and pseudo labels for the missing ones. Importantly, data without any ground-truth labels can also be involved in our framework, as we can regard them as data with all labels missing and generate the corresponding pseudo labels. We carefully re-organized multiple publicly available nucleus classification datasets, converted them into a uniform format, and tested the proposed framework on them. Experimental results show substantial improvement compared to the state-of-the-art methods. The code and data are available at https://w-h-zhang.github.io/projects/dataset_merging/dataset_merging.html.

HCFPN: Hierarchical Contextual Feature-Preserved Network for Remote Sensing Scene Classification

Convolutional neural networks (CNNs) have made significant advances in remote sensing scene classification (RSSC) in recent years. Nevertheless, the limitations of the receptive field cause CNNs to suffer from a disadvantage in capturing contextual information. To address this issue, vision transformer (ViT), a novel model that has piqued the interest of academics, is used to extract latent contextual information in remote sensing scene classification. However, when confronted with the challenges of large-scale variations and high interclass similarity in scene classification images, the original ViT has the drawback of ignoring important local features, thereby causing the model’s performance to degrade. Consequently, we propose the hierarchical contextual feature-preserved network (HCFPN) by combining the advantages of CNNs and ViT. First, a hierarchical feature extraction module based on ResNet-34 is utilized to acquire the multilevel convolutional features and high-level semantic features. Second, a contextual feature-preserved module takes advantage of the first two multilevel features to capture abundant long-term contextual features. Then, the captured long-term contextual features are utilized for multiheaded cross-level attention computing to aggregate and explore the correlation of multilevel features. Finally, the multiheaded cross-level attention score and high-level semantic features are classified. Then, a category score average module is proposed to fuse the classification results, whereas a label smoothing approach is utilized prior to calculating the loss to produce discriminative scene representation. In addition, we conduct extensive experiments on two publicly available RSSC datasets. Our proposed HCPFN outperforms most state-of-the-art approaches.

Self-supervised visual representation learning on food images

Food image classification is the groundwork for image-based dietary assessment, which is the process of monitoring what kinds of food and how much energy is consumed using captured food or eating scene images. Existing deep learning based methods learn the visual representation for food classification based on human annotation of each food image. However, most food images captured from real life are obtained without labels, requiring human annotation to train deep learning based methods. This approach is not feasible for real world deployment due to high costs. To make use of the vast amount of unlabeled images, many existing works focus on unsupervised or self-supervised learning to learn the visual representation directly from unlabeled data. However, none of these existing works focuses on food images, which is more challenging than general objects due to its high inter-class similarity and intra-class variance. In this paper, we focus on two items: the comparison of existing models and the development of an effective self-supervised learning model for food image classification. Specifically, we first compare the performance of existing state-of-the-art self-supervised learning models, including SimSiam, SimCLR, SwAV, BYOL, MoCo, and Rotation Pretext Task on food images. The experiments are conducted on the Food-101 dataset, which contains 101 different classes of foods with 1,000 images in each class. Next, we analyze the unique features of each model and compare their performance on food images to identify the key factors in each model that can help improve the accuracy. Finally, we propose a new model for unsupervised visual representation learning on food images for the classification task.

Few-shot incremental learning with continual prototype calibration for remote sensing image fine-grained classification

With the rapid acquisition of remote sensing (RS) data, new categories of objects continue to emerge, and some categories can only obtain a few training samples. Thus, few-shot class-incremental learning (FSCIL) has received intense attention in recent years. Nevertheless, the existing FSCIL methods are difficult to adapt to RS characteristics with high inter-class similarity, large intra-class differences, and complex backgrounds. In this paper, we propose the FSCIL method called continual prototype calibration (CPC), which applies the decoupled learning strategy to cope with the demand to retrain the model when new few-shot categories appear. It can also address the tedious problems in fine-grained RS images by calibrating inter-class and intra-class prototypes. Concretely, considering the severe issue of inter-class confusion, we introduce a prototype separability module (PSM) to update the distribution of inter-class prototypes. In this way, we can distinguish different categories more clearly by weakening their similarities, and the phenomenon of catastrophic forgetting can also be effectively mitigated. Furthermore, to generate more representative and accurate intra-class prototypes, we design a Meta-Network based on foreground enhancement mechanism (FEM). The Meta-Network can boost the learning ability of the model for few-shot data and suppress the occurrence of overfitting by training on task-based data. And the FEM enhances the object features by filtering out redundant background information, facilitating fine-grained classification. Our CPC method is the first FSCIL method applied to optical RS images, and we verify its effectiveness on three datasets. Extensive experiments illustrate that our method achieves state-of-the-art performance in FSCIL of fine-grained RS images. The code is available at https://github.com/ningerhhh/CPC.