Masked Images Research Articles

One-shot segmentation of novel white matter tracts via extensive data augmentation and adaptive knowledge transfer.

The use of convolutional neural networks (CNNs) has allowed accurate white matter (WM) tract segmentation on diffusion magnetic resonance imaging (dMRI). To train the CNN-based segmentation models, a large number of scans on which WM tracts are annotated need to be collected, and these annotated scans can be accumulated over a long period of time. However, when novel WM tracts that are different from existing annotated WM tracts are of interest, additional annotations are required for their segmentation. Due to the cost of manual annotations, methods have been developed for few-shot segmentation of novel WM tracts, where the segmentation knowledge is transferred from existing WM tracts to novel WM tracts and the amount of annotated data for novel WM tracts is reduced. Despite these developments, it is desirable to further reduce the amount of annotated data to the one-shot setting with a single annotated image. To address this problem, we develop an approach to one-shot segmentation of novel WM tracts. Our method follows the existing pretraining/fine-tuning framework that transfers segmentation knowledge from existing to novel WM tracts. First, as there is extremely scarce annotated data in the one-shot setting, we design several different data augmentation strategies so that extensive data augmentation can be performed to obtain extra synthetic training data. The data augmentation strategies are based on image masking and thus applicable to the one-shot setting. Second, to address overfitting and knowledge forgetting in the fine-tuning stage that can be more severe given limited training data, we propose an adaptive knowledge transfer strategy that selects the network weights to be updated. The data augmentation and adaptive knowledge transfer strategies are combined to train the segmentation model. Considering that the different data augmentation strategies can generate synthetic data that contain potentially conflicting information, we apply the data augmentation strategies separately, each leading to a different segmentation model. The results predicted by the different models are fused to produce the final segmentation. We validated our method on two brain dMRI datasets, including a public dataset and an in-house dataset. Different settings were considered for the validation, and the results show that the proposed method improves the one-shot segmentation of novel WM tracts.

Self-Supervised Triplet Contrastive Learning for Classifying Endometrial Histopathological Images.

Early identification of endometrial cancer or precancerous lesions from histopathological images is crucial for precise endometrial medical care, which however is increasing hampered by the relative scarcity of pathologists. Computer-aided diagnosis (CAD) provides an automated alternative for confirming endometrial diseases with either feature-engineered machine learning or end-to-end deep learning (DL). In particular, advanced self-supervised learning alleviates the dependence of supervised learning on large-scale human-annotated data and can be used to pre-train DL models for specific classification tasks. Thereby, we develop a novel self-supervised triplet contrastive learning (SSTCL) model for classifying endometrial histopathological images. Specifically, this model consists of one online branch and two target branches. The second target branch includes a simple yet powerful augmentation module named random mosaic masking (RMM), which functions as an effective regularization by mapping the features of masked images close to those of intact ones. Moreover, we add a bottleneck Transformer (BoT) model into each branch as a self-attention module to learn the global information by considering both content information and relative distances between features at different locations. On public endometrial dataset, our model achieved four-class classification accuracies of 77.31 ± 0.84, 80.87 ± 0.48 and 83.22 ± 0.87% using 20, 50 and 100% labeled images, respectively. When transferred to the in-house dataset, our model obtained a three-class diagnostic accuracy of 96.81% with 95% confidence interval of 95.61-98.02%. On both datasets, our model outperformed state-of-the-art supervised and self-supervised methods. Our model may help pathologists to automatically diagnose endometrial diseases with high accuracy and efficiency using limited human-annotated histopathological images.

AutoSMIM: Automatic Superpixel-Based Masked Image Modeling for Skin Lesion Segmentation.

Skin lesion segmentation from dermoscopic images plays a vital role in early diagnoses and prognoses of various skin diseases. However, it is a challenging task due to the large variability of skin lesions and their blurry boundaries. Moreover, most existing skin lesion datasets are designed for disease classification, with relatively fewer segmentation labels having been provided. To address these issues, we propose a novel automatic superpixel-based masked image modeling method, named autoSMIM, in a self-supervised setting for skin lesion segmentation. It explores implicit image features from abundant unlabeled dermoscopic images. autoSMIM begins with restoring an input image with randomly masked superpixels. The policy of generating and masking superpixels is then updated via a novel proxy task through Bayesian Optimization. The optimal policy is subsequently used for training a new masked image modeling model. Finally, we finetune such a model on the downstream skin lesion segmentation task. Extensive experiments are conducted on three skin lesion segmentation datasets, including ISIC 2016, ISIC 2017, and ISIC 2018. Ablation studies demonstrate the effectiveness of superpixel-based masked image modeling and establish the adaptability of autoSMIM. Comparisons with state-of-the-art methods show the superiority of our proposed autoSMIM. The source code is available at https://github.com/Wzhjerry/autoSMIM.

Property-microstructure analysis and statistical qualification of MgB2 bulks: An investigation via micrograph feature extraction

In the pursuit of high-quality superconductors for stable operation at cryogenic temperatures, the understanding of process-structure-property relationships is crucial to device performance reproducibility and scalability. In this manuscript, we propose an indirect image-based method to evaluate statistical relationships between the ex-situ characterization results (i.e., superconducting, and mechanical properties) and microstructural distribution of MgB2 bulks produced via Spark Plasma Sintering. This study utilizes bulks processed under similar temperature profiles and sintering time, but different uniaxial loads and powder modalities. Considering the microstructure as the centerpiece between the MgB2 sintering thermodynamics and yielded macroscopic properties, multiple cross-sectional SEM micrographs of bulks MgB2 undergo an automated process that comprise image thresholding, segmentation, masking, and filtering to enable the identification of microstructural defects and byproducts. The statistical analysis establishes the relationship between processing conditions, bulk microstructural distributions, and ex-situ characterization results. Lastly, morphological analysis of the bulk cross-section microstructures indicates the effectiveness of the proposed method in identifying and statistically quantifying property variation in MgB2 bulk properties.

Multi-View Masked Autoencoder for General Image Representation

Self-supervised learning is a method that learns general representation from unlabeled data. Masked image modeling (MIM), one of the generative self-supervised learning methods, has drawn attention for showing state-of-the-art performance on various downstream tasks, though it has shown poor linear separability resulting from the token-level approach. In this paper, we propose a contrastive learning-based multi-view masked autoencoder for MIM, thus exploiting an image-level approach by learning common features from two different augmented views. We strengthen the MIM by learning long-range global patterns from contrastive loss. Our framework adopts a simple encoder–decoder architecture, thus learning rich and general representations by following a simple process: (1) Two different views are generated from an input image with random masking and by contrastive loss, we can learn the semantic distance of the representations generated by an encoder. By applying a high mask ratio, of 80%, it works as strong augmentation and alleviates the representation collapse problem. (2) With reconstruction loss, the decoder learns to reconstruct an original image from the masked image. We assessed our framework through several experiments on benchmark datasets of image classification, object detection, and semantic segmentation. We achieved 84.3% in fine-tuning accuracy on ImageNet-1K classification and 76.7% in linear probing, thus exceeding previous studies and showing promising results on other downstream tasks. The experimental results demonstrate that our work can learn rich and general image representation by applying contrastive loss to masked image modeling.

HEXA-GAN: Skin lesion image inpainting via hexagonal sampling based generative adversarial network

Skin cancer is the uncontrollable proliferation of abnormal cells in the epidermis. In manual examination, the dermoscopic images are used by clinicians to detect skin cancer. However, digital hair removal (DHR) is more challenging task due to the presence of the hair occlusion that disturbs the intrinsic intensities and geometric features of the lesion regions that negatively affect diagnosis. Some of the dermis hairs in dermoscopic images are thicker, thinner, overlaps, fades, and obscures or covers textural changes. To overcome these challenges, a novel deep learning based Hexa-GAN model is proposed for hair segmentation and hair gap inpainting. Initially, the images from ISIC dataset are given to Attention U-net for segmenting the hairs in pixel units. The masked hair gap images from Attention U-net are given to the proposed Hexa-GAN with ReLU activation function. The first GAN (G1) performs coarse inpainting and reconstructs images using hexagonal sampling. Hexagonal divisions are merged into coarse inpainted images. The output of G1 is fed into the second GAN (G2) for fine reconstruction with contextual attention layer to generate the missing patches of the image. Finally, the output from G2 is merged into the original inpainted images without hair regions. From the experimental analysis, the Attention-Unet achieves better performance based on Jaccard index of 94.64% and Dice Index of 82.61% respectively. The proposed Hexa-GAN model increases the average PSNR range by 9.89%, 3.83%, 2.75%, 5.68%, 4.87% and 6.11% compared to CHC-Otsu model, Encoder-Decoder CNN, SN-PatchGAN, Variational Autoencoder, AlexNet architecture and FCN8-ResNetC approach respectively.

Eye-Gaze-Guided Vision Transformer for Rectifying Shortcut Learning.

Learning harmful shortcuts such as spurious correlations and biases prevents deep neural networks from learning meaningful and useful representations, thus jeopardizing the generalizability and interpretability of the learned representation. The situation becomes even more serious in medical image analysis, where the clinical data are limited and scarce while the reliability, generalizability and transparency of the learned model are highly required. To rectify the harmful shortcuts in medical imaging applications, in this paper, we propose a novel eye-gaze-guided vision transformer (EG-ViT) model which infuses the visual attention from radiologists to proactively guide the vision transformer (ViT) model to focus on regions with potential pathology rather than spurious correlations. To do so, the EG-ViT model takes the masked image patches that are within the radiologists' interest as input while has an additional residual connection to the last encoder layer to maintain the interactions of all patches. The experiments on two medical imaging datasets demonstrate that the proposed EG-ViT model can effectively rectify the harmful shortcut learning and improve the interpretability of the model. Meanwhile, infusing the experts' domain knowledge can also improve the large-scale ViT model's performance over all compared baseline methods with limited samples available. In general, EG-ViT takes the advantages of powerful deep neural networks while rectifies the harmful shortcut learning with human expert's prior knowledge. This work also opens new avenues for advancing current artificial intelligence paradigms by infusing human intelligence.

Gender recognition in masked facial images using EfficientNet and transfer learning approach

Gender recognition in masked facial images using EfficientNet and transfer learning approach

Self-supervised learning with self-distillation on COVID-19 medical image classification

Background and objectiveCurrently, COVID-19 is a highly infectious disease that can be clinically diagnosed based on diagnostic radiology. Deep learning is capable of mining the rich information implied in inpatient imaging data and accomplishing the classification of different stages of the disease process. However, a large amount of training data is essential to train an excellent deep-learning model. Unfortunately, due to factors such as privacy and labeling difficulties, annotated data for COVID-19 is extremely scarce, which encourages us to propose a more effective deep learning model that can effectively assist specialist physicians in COVID-19 diagnosis. MethodsIn this study,we introduce Masked Autoencoder (MAE) for pre-training and fine-tuning directly on small-scale target datasets. Based on this, we propose Self-Supervised Learning with Self-Distillation on COVID-19 medical image classification (SSSD-COVID). In addition to the reconstruction loss computation on the masked image patches, SSSD-COVID further performs self-distillation loss calculations on the latent representation of the encoder and decoder outputs. The additional loss calculation can transfer the knowledge from the global attention of the decoder to the encoder which acquires only local attention. ResultsOur model achieves 97.78 % recognition accuracy on the SARS-COV-CT dataset containing 2481 images and is further validated on the COVID-CT dataset containing 746 images, which achieves 81.76 % recognition accuracy. Further introduction of external knowledge resulted in experimental accuracies of 99.6% and 95.27 % on these two datasets, respectively. ConclusionsSSSD-COVID can obtain good results on the target dataset alone, and when external information is introduced, the performance of the model can be further improved to significantly outperform other models.Overall, the experimental results show that our method can effectively mine COVID-19 features from rare data and can assist professional physicians in decision-making to improve the efficiency of COVID-19 disease detection.

Machine learning algorithms for lithological mapping using Sentinel-2 and SRTM DEM in highly vegetated areas

Lithological mapping in highly vegetated areas using remote sensing techniques poses a significant challenge. Inspired by the concept of “geobotany”, we attempted to distinguish lithologies indirectly using machine learning algorithms (MLAs) based on Sentinel-2 and SRTM DEM in Zhangzhou City, Fujian Province. The study area has high vegetation cover, with lithologies that are largely obscured. After preprocessing such as cloud masking, resampling, and median image synthesis, 17 spectral bands and features from Sentinel-2 and 9 terrain features from DEM were extracted. Five widely used MLAs, MD, CART, SVM, RF, and GBDT, were trained and validated for lithological mapping. The results indicate that advanced MLAs, such as GBDT and RF, are highly effective for nonlinear modeling and learning with relative increases reaching 8.18%∼11.82% for GBDT and 6.36%∼10% for RF. Compared with optical imagery or terrain data alone, combining Sentinel-2 and DEM significantly improves the accuracy of lithological mapping, as it provides more comprehensive and precise spectral characteristics and spatial information. GBDT_Sen+DEM utilizing integrated data achieved the highest classification accuracy, with an overall accuracy of 63.18%. This study provides a case study for lithological mapping of areas with high vegetation cover at the local level. This also reinforces the idea that merging remote sensing and terrain data significantly enhances the precision and reliability of the lithological mapping methods.

Unsupervised domain adaptation-based crack segmentation using transformer network

Surface cracks are a common structural defect. The intelligent inspection of these defects through computer vision and deep learning is of paramount importance for early maintenance and operation. Despite the remarkable success of supervised learning methods in detecting surface cracks, their performance heavily relies on the availability of extensive labeled datasets. Annotating a single image can be a time-consuming process, prone to human error. Moreover, these methods often struggle to generalize effectively to unseen datasets due to disparities between source and target images. To address this issue, unsupervised domain adaptation comes into play, as it aims to transfer knowledge learned from the labeled source domain to the unlabeled target domain. Consequently, we conducted an evaluation of a recent unsupervised domain adaptation model for semantic segmentation that incorporates masked image consistency into DAFormer, a state-of-the-art model with the ability to adapt to various datasets. To assess the model’s performance, we employed three publicly available crack datasets, each containing background and crack classes. Our study has revealed that : (1) SegFormer, a transformer-based model, outperforms ConvNet-based models without utilizing adaptation knowledge, demonstrating superior generalizability to previously unseen data. (2) The unsupervised domain-adaptation model consistently outperforms the source model, resulting in a significant enhancement in the mean intersection over union of SegFormer’s source-only approach by a remarkable 10% to 22%. With the exception of a single case, the relative performance of unsupervised domain adaptation compared to supervised training with labeled data exceeds 85%, underscoring its promising performance in crack segmentation. Consequently, our adopted method emerges as a viable alternative, particularly in scenarios where labeled data is scarce or prohibitively expensive.

MaskDUF: Data uncertainty learning in masked face recognition with mask uncertainty fluctuation

As an essential component of experts and intelligent systems, Masked Face Recognition (MFR) has been applied to various applications, but it is still a challenging task due to the aggravated uncertainty caused by mask occlusion. Most existing MFR methods are deterministic point embedding models that are limited in representing the uncertainty in masked face images. Data Uncertainty Learning (DUL) is an advanced uncertainty modeling method, but there are two problems when it is applied to MFR task: (1) the masked face tends to be regarded as noise due to the mask occlusion, which weakens its optimization; (2) the large representation difference between face and masked face results in a dispersed intra-class distribution. To solve the above problems, we propose a novel masked face data uncertainty learning method (MaskDUF) for MFR task, which can adaptively adjust the optimization weight by modeling the uncertainty and measuring the recognizability of samples, thus learning an ideal sample distribution with compact intra-class, discrepant inter-class and distant noise. Specifically, a Hard Kullback–Leibler Divergence (H-KLD) method is proposed to serve as an adaptive variance regularizer for masked faces, which contributes to learning more accurate uncertainty representations and avoiding overfitting noise. Moreover, by combining feature magnitude and variance uncertainty, Mask Uncertainty Fluctuation (MUF) is proposed to comprehensively measure sample recognizability, which contributes to enhancing the learning preference of masked faces and constructing a more compact cone-like intra-class distribution. Finally, compared with other advanced models, MaskDUF achieves an average accuracy improvement of 1.33% to 13.28%, and its effectiveness and strong robustness are also proved by ablation study, noise experiment and parametric analysis.

Automated Platelet Counter with Detection Using K-Means Clustering

Platelet is a blood cell type that is stored and circulated in the human body. It acts as a blood thickening agent and prevents blood from overflowing whenever bleeding occurs. An excessive or inadequate number of platelets could lead to platelet-related diseases. The current practice of platelet counting involves the manual counting process using a haemocytometer, Wright’s Stain which uses the dyes to facilitate the differentiation of blood cell types, and a tally counter. Yet, this process can be time-consuming, demanding, and exhausting for haematologists, and likely to be prone to errors. Thus, this paper presents a study on automated platelet counter and detection using image processing techniques. The K-Means Clustering was employed to count and detect the presence of platelets in microscopic blood smear images. Several processes were performed prior to the K-means clustering, including image enhancement and YCbCr image formatting. Subsequently, image masking, as well as area thresholding were applied to eliminate every unwanted entity and highlight the visibility of the platelets before the number of platelets could be detected and counted. A comparative experiment was designed in which the K-Means Clustering platelet count and detection were compared with the actual number of platelets reported by haematologists. The platelet counts and detection were categorized into three detection categories which are Less Detection (LD), Accurate Detection (AD), and Over Detection (OD). The proposed study was evaluated to 90 testing platelet images. Out of the 90 testing images, 75 platelet images were perfectly counted and detected which returned 91.67% of accuracy. This signifies that the K-Means Clustering algorithm was discovered to be efficient and dependable for automated platelet counter and detection

Face Mask Image Classification Using Fine-Tuning and The Effect of FGSM And PGD Attacks.

This research seeks to use a fine-tuning method for automatic diagnostics and classification of face masks without face mask images. We trained Deep Convolutional Neural Network models on the face mask dataset obtained from the Mendeley data repository to determine whether individuals will adhere to policies that face mask reduces the spread of SARS-COV-2. The proposed architectures used in this study include the VGG19, MobileNetV3, and InceptionV3 models. These models are known for their role in classifying images. They were trained using the fine-tuning approach and their respective outputs were compared. After training the face mask dataset, it can be said that the fine-tuned InceptionV3 model performed extremely well against the other models obtaining an accuracy of 99.21%, and was able to predict 99.63% of the test dataset. However, the robustness of the fine-tuned model was tested against fast gradient sign method (FGSM) and projected gradient descent (PGD) attacks which generate adversarial images using the gradients of the model. Additionally, the classification report shows after the FGSM attack that the model's accuracy was reduced by 43%. Also, after the PDG attack, the accuracy of the model was reduced by 19%. We assessed the models using many performance metrics such as precision, recall, F1 score, and accuracy before subjecting them to two common adversarial attack techniques: FGSM and PDG. Finally, we demonstrated how the proposed robust method improved the model’s defense against adversarial attacks. The findings emphasize the critical need to increase awareness about adversarial assaults on SARS-COV-2 monitoring systems and to advocate for proactive steps to protect healthcare systems from comparable threats prior to practical deployment.

Deep Learning-Based Esophageal Tumor Identification on CT Slice

The study aims to use deep learning to improve the accuracy of identifying esophageal tumors on CT slices for radiotherapy planning. The identification of Gross Tumor Volume (GTV) can be challenging due to low contrast with surrounding tissue. Other methods like endoscopy and PET scan can provide additional information, but may not be suitable for radiotherapy due to differences in tissue density and alignment needs. A multi-task deep learning network was developed, which perform both segmentation and slice classification, simultaneously. For segmentation, the normal esophagus and tumor were treated as a single structure due to the difficulty in distinguishing between them on CT images. The slices were divided into 3 categories, including tumor, normal esophageal and other. An Unet was used for segmentation and generate the mask to remove irrelevant areas. The masked image will be input into a Resnet to obtain the categories of slices. The performance of classification was accessed by ROC curve, AUC and confusion matrix on a new dataset and PET images. The multi-task deep learning network was developed on a dataset of 315 patients' CT images and GTV segmentations, which were reviewed and verified by physicians. The model was then evaluated on an additional validation dataset of 30 patients, resulting in an accuracy of 88%. In terms of sensitivity and specificity, the model showed high performance, with a sensitivity of 97% and specificity of 95% for tumor and normal esophagus in the validation dataset. Meanwhile, the specificity was 85% and the specificity was 80% for tumor and normal esophagus in PET images dataset. This multi-task deep learning approach effectively combines the benefits of both segmentation and classification techniques, resulting in improved accuracy and efficiency in identifying esophageal tumors on CT slices for radiotherapy planning.

High-Quality Object Detection Method for UAV Images Based on Improved DINO and Masked Image Modeling

The extensive application of unmanned aerial vehicle (UAV) technology has increased academic interest in object detection algorithms for UAV images. Nevertheless, these algorithms present issues such as low accuracy, inadequate stability, and insufficient pre-training model utilization. Therefore, a high-quality object detection method based on a performance-improved object detection baseline and pretraining algorithm is proposed. To fully extract global and local feature information, a hybrid backbone based on the combination of convolutional neural network (CNN) and vision transformer (ViT) is constructed using an excellent object detection method as the baseline network for feature extraction. This backbone is then combined with a more stable and generalizable optimizer to obtain high-quality object detection results. Because the domain gap between natural and UAV aerial photography scenes hinders the application of mainstream pre-training models to downstream UAV image object detection tasks, this study applies the masked image modeling (MIM) method to aerospace remote sensing datasets with a lower volume than mainstream natural scene datasets to produce a pre-training model for the proposed method and further improve UAV image object detection accuracy. Experimental results for two UAV imagery datasets show that the proposed method achieves better object detection performance compared to state-of-the-art (SOTA) methods with fewer pre-training datasets and parameters.

Confidence-guided mask learning for semi-supervised medical image segmentation

Semi-supervised learning aims to train a high-performance model with a minority of labeled data and a majority of unlabeled data. Existing methods mostly adopt the mechanism of prediction task to obtain precise segmentation maps with the constraints of consistency or pseudo-labels, whereas the mechanism usually fails to overcome confirmation bias. To address this issue, in this paper, we propose a novel Confidence-Guided Mask Learning (CGML) for semi-supervised medical image segmentation. Specifically, on the basis of the prediction task, we further introduce an auxiliary generation task with mask learning, which intends to reconstruct the masked images for extremely facilitating the model capability of learning feature representations. Moreover, a confidence-guided masking strategy is developed to enhance model discrimination in uncertain regions. Besides, we introduce a triple-consistency loss to enforce a consistent prediction of the masked unlabeled image, original unlabeled image and reconstructed unlabeled image for generating more reliable results. Extensive experiments on two datasets demonstrate that our proposed method achieves remarkable performance.

Diabetic Retinopathy Screening Using Smartphone-Based Fundus Photography and Deep-Learning Artificial Intelligence in the Yucatan Peninsula: A Field Study.

To compare the performance of Medios (offline) and EyeArt (online) artificial intelligence (AI) algorithms for detecting diabetic retinopathy (DR) on images captured using fundus-on-smartphone photography in a remote outreach field setting. In June, 2019 in the Yucatan Peninsula, 248 patients, many of whom had chronic visual impairment, were screened for DR using two portable Remidio fundus-on-phone cameras, and 2130 images obtained were analyzed, retrospectively, by Medios and EyeArt. Screening performance metrics also were determined retrospectively using masked image analysis combined with clinical examination results as the reference standard. A total of 129 patients were determined to have some level of DR; 119 patients had no DR. Medios was capable of evaluating every patient with a sensitivity (95% confidence intervals [CIs]) of 94% (88%-97%) and specificity of 94% (88%-98%). Owing primarily to photographer error, EyeArt evaluated 156 patients with a sensitivity of 94% (86%-98%) and specificity of 86% (77%-93%). In a head-to-head comparison of 110 patients, the sensitivities of Medios and EyeArt were 99% (93%-100%) and 95% (87%-99%). The specificities for both were 88% (73%-97%). Medios and EyeArt AI algorithms demonstrated high levels of sensitivity and specificity for detecting DR when applied in this real-world field setting. Both programs should be considered in remote, large-scale DR screening campaigns where immediate results are desirable, and in the case of EyeArt, online access is possible.

RemainNet: Explore Road Extraction from Remote Sensing Image Using Mask Image Modeling

Road extraction from a remote sensing image is a research hotspot due to its broad range of applications. Despite recent advancements, achieving precise road extraction remains challenging. Since a road is thin and long, roadside objects and shadows cause occlusions, thus influencing the distinguishment of the road. Masked image modeling reconstructs masked areas from unmasked areas, which is similar to the process of inferring occluded roads from nonoccluded areas. Therefore, we believe that mask image modeling is beneficial for indicating occluded areas from other areas, thus alleviating the occlusion issue in remote sensing image road extraction. In this paper, we propose a remote sensing image road extraction network named RemainNet, which is based on mask image modeling. RemainNet consists of a backbone, image prediction module, and semantic prediction module. An image prediction module reconstructs a masked area RGB value from unmasked areas. Apart from reconstructing original remote sensing images, a semantic prediction module of RemainNet also extracts roads from masked images. Extensive experiments are carried out on the Massachusetts Roads dataset and DeepGlobe Road Extraction dataset; the proposed RemainNet improves 0.82–1.70% IoU compared with other state-of-the-art road extraction methods.

S3Mix: Same Category Same Semantics Mixing for Augmenting Fine-grained Images

Data augmentation is a common technique to improve the generalization performance of models for image classification. Although methods such as Mixup and CutMix that mix images randomly are indeed instrumental in general image classification, randomly swapping or masking regions is not friendly to fine-grained images, since the key to fine-grained image classification precisely lies in discriminative and informative regions, and it is unreasonable to generate labels solely consistent with the proportion of synthesis. Some erasing methods like Cutout even endanger fine-grained image classification because of erasing the discriminative regions by chance. In this article, we propose the Same Category Same Semantics Mixing method (S3Mix) corresponding to the characteristics of fine-grained images. Specifically, we limit the mixture to regions of the same category and semantics. The core of the method is two constraints. The exchange with the semantic region ensures the discrimination and semantics integrity of the generated image, and the exchange in the same class avoids the problem of unreasonable label generation. At the same time, we propose a homology loss to promote the semantic relationship between the generated positive image pairs. Experiments have been conducted on four fine-grained datasets, and the results show the proposed method is superior to the traditional image augmentation methods as well as some fine-grained data augmentation methods.