Self-supervised Learning Research Articles

Industrial Image Anomaly Detection via Self-Supervised Learning with Feature Enhancement Assistance

Industrial anomaly detection is constrained by the scarcity of anomaly samples, limiting the applicability of supervised learning methods. Many studies have focused on anomaly detection by generating anomaly images and adopting self-supervised learning approaches. Leveraging pre-trained networks on ImageNet has been explored to assist in this training process. However, achieving accurate anomaly detection remains time-consuming due to the network’s depth and parameter count not being reduced. In this paper, we propose a self-supervised learning method based on Feature Enhancement Patch Distribution Modeling (FEPDM), which generates simulated anomalies. Unlike direct training on the original feature extraction network, our approach utilizes a pre-trained network to extract multi-scale features. By aggregating these multi-scale features, we are able to train at the feature level, thereby adapting more efficiently to various network structures and reducing domain bias with respect to natural image classification. Additionally, it significantly reduces the number of parameters in the training process. Introducing this approach not only enhances the model’s generalization ability but also significantly improves the efficiency of anomaly detection. The method was evaluated on MVTec AD and BTAD datasets, and (image-level, pixel-level) AUROC scores of (95.7%, 96.2%), (93.4%, 97.6%) were obtained, respectively. The experimental results have convincingly demonstrated the efficacy of our method in tackling the scarcity of abnormal samples in industrial scenarios, while simultaneously highlighting its broad generalizability.

Efficiency-oriented approaches for self-supervised speech representation learning

Efficiency-oriented approaches for self-supervised speech representation learning

Correction to: Mitigating selection bias in counterfactual prediction through self-supervised domain embedding learning with virtual samples

Correction to: Mitigating selection bias in counterfactual prediction through self-supervised domain embedding learning with virtual samples

Robust seismic data denoising via self-supervised deep learning

Seismic data denoising is a critical component of seismic data processing, yet effectively removing erratic noise, characterized by its non-Gaussian distribution and high amplitude, remains a substantial challenge for conventional methods and deep-learning (DL) algorithms. Supervised learning frameworks typically outperform others, but they require pairs of noisy data sets alongside corresponding clean ground truth, which is impractical for real-world seismic data sets. In contrast, unsupervised learning (UL) methods, which do not rely on ground truth during training, often fall short in performance when compared with their supervised or traditional denoising counterparts. Moreover, current unsupervised DL methods fail to address the specific challenges posed by erratic seismic noise adequately. This paper introduces a novel zero-shot unsupervised DL framework designed specifically to mitigate random and erratic noise, with a particular emphasis on blended noise. Drawing inspiration from Noise2Noise (N2N) and data augmentation principles, we develop a robust self-supervised denoising network called robust Noiser2Noiser. Our approach eliminates the need for paired noisy and clean data sets as required by supervised methods or paired noisy data sets as in N2N. Instead, our framework relies solely on the original noisy seismic data set. Our methodology generates two independent recorrupted data sets from the original noisy data set, using one as the input and the other as the training target. Subsequently, we use a DL-based denoiser, denoising convolutional neural network, for training purposes. To address various types of random and erratic noise, the original noisy data set is recorrupted with the same noise type. Detailed explanations for generating training input and target data for blended data are provided. We apply our network to synthetic and real marine data examples, demonstrating significantly improved noise attenuation performance compared with traditional denoising methods and state-of-the-art UL Codes are available on https://github.com/Ji-seismic/N2N_deblending .

SelfMixed: Self-supervised mixed noise attenuation for distributed acoustic sensing data

Distributed acoustic sensing (DAS) is an emerging data acquisition technique known for its high sensing density, cost effectiveness, and environmental friendliness, making it a technology with significant future application potential in many fields. However, DAS signals are often contaminated by various types of noise, such as high-frequency, high-amplitude erratic, and horizontal noise, making their processing challenging. Therefore, it is crucial to leverage the physical characteristics of these diverse types of noise in DAS data and effectively attenuate them. In this work, we develop SelfMixed, a novel self-supervised learning method for mixed noise suppression of DAS data. We fully exploit the physical characteristics of different types of noise in DAS data and introduce a physical characteristic-based training strategy. Specifically, we use the [Formula: see text] norm to characterize random noise, the [Formula: see text] norm for erratic noise, and horizontal smoothness and vertical nonsmoothness for horizontal noise. In addition, we use a blind-spot-based training strategy for DAS denoising, relying solely on observed noisy data. To more effectively attenuate horizontal noise, we also introduce a Fourier transform-based parameterization method. By combining self-supervised deep priors with the physical characteristics of mixed DAS noise, our method effectively attenuates complex mixed noise in field DAS data. Extensive experiments on synthetic and field data from various geographic scenarios validate the superiority of SelfMixed over seven state-of-the-art DAS denoising approaches.

PMANet: Malicious URL detection via post-trained language model guided multi-level feature attention network

The expansion of the Internet has led to the widespread proliferation of malicious URLs, becoming a primary vector for cyber threats. Detecting malicious URLs is now essential for improving network security. The technological revolution spurred by pre-trained language models holds great promise for advancing the detection of malicious URLs. However, current research applying these models to URLs fails to address several crucial factors, including the lack of domain-specific adaptability, the omission of character-level information, and the neglect of both local detail extraction and low-order encoding information. In this paper, we propose PMANet, a pre-trained Language Model-Guided multi-level feature attention network, for addressing these issues. To facilitate a smooth transition of the pre-trained Transformer into the URL domain and to enable it to effectively capture information at both subword and character levels, we propose a post-training program that continues training the model on URLs using three self-supervised learning objectives: masked language model, noisy language model, and domain discrimination task. Subsequently, we develop a module to capture the output of each encoding layer, thus extracting hierarchical representations of URLs spanning from low-level to high-level. In addition, we propose a layer-wise attention mechanism that dynamically assigns weight coefficients to these feature layers based on their relevance. Finally, we apply spatial pyramid pooling to perform multi-scale down-sampling in order to obtain both local features and global context. PMANet achieves multifaceted integration in URL feature extraction, including capturing information at both the lexical and character levels, extracting features from low to high order, and discerning patterns at both global and local scales. We evaluate PMANet against challenging real-world scenarios, such as small-scale data, class imbalance, cross-dataset, adversarial attacks, and case studies on active malicious URLs. All experiments demonstrate that PMANet exhibits superiority over both the previous state-of-the-art pre-trained models and conventional deep learning models. Specifically, PMANet still achieves a 0.9941 AUC under adversarial attacks and correctly identifies all 20 actively malicious URLs in the case study. The code and data for our research are available at: https://github.com/Alixyvtte/Malicious-URL-Detection-PMANet.

Rethinking masked image modelling for medical image representation

Masked Image Modelling (MIM), a form of self-supervised learning, has garnered significant success in computer vision by improving image representations using unannotated data. Traditional MIMs typically employ a strategy of random sampling across the image. However, this random masking technique may not be ideally suited for medical imaging, which possesses distinct characteristics divergent from natural images. In medical imaging, particularly in pathology, disease-related features are often exceedingly sparse and localized, while the remaining regions appear normal and undifferentiated. Additionally, medical images frequently accompany reports, directly pinpointing pathological changes’ location. Inspired by this, we propose Masked medical Image Modelling (MedIM), a novel approach, to our knowledge, the first research that employs radiological reports to guide the masking and restore the informative areas of images, encouraging the network to explore the stronger semantic representations from medical images. We introduce two mutual comprehensive masking strategies, knowledge-driven masking (KDM), and sentence-driven masking (SDM). KDM uses Medical Subject Headings (MeSH) words unique to radiology reports to identify symptom clues mapped to MeSH words (e.g., cardiac, edema, vascular, pulmonary) and guide the mask generation. Recognizing that radiological reports often comprise several sentences detailing varied findings, SDM integrates sentence-level information to identify key regions for masking. MedIM reconstructs images informed by this masking from the KDM and SDM modules, promoting a comprehensive and enriched medical image representation. Our extensive experiments on seven downstream tasks covering multi-label/class image classification, pneumothorax segmentation, and medical image–report analysis, demonstrate that MedIM with report-guided masking achieves competitive performance. Our method substantially outperforms ImageNet pre-training, MIM-based pre-training, and medical image–report pre-training counterparts. Codes are available at https://github.com/YtongXie/MedIM.

State-of-the-Art and Future Directions

In our daily lives, smart AI-powered devices including computers are becoming more common. But the problem is that sometimes these AI systems make decisions that seem like magic, making it hard for us to know how they work and also, we are not able to predict the future (like after making a decision, we are not able to predict whether the result will be positive or negative). This research paper is all about fixing this issue and making the AI strong enough to make a decision and learn each positive and negative thing from the surroundings. We dive into the world of Explainable Artificial Intelligence (XAI), finding new and simple ways to make complex AI systems easy for people to understand. In this Research paper, we are also going to make the AI that much more powerful so that it can learn itself and gather information from its surroundings. This would be a fantastic step for AI to learn Patterns, Structures, and Unlabeled data allowing it to identify hidden relationships and features without explicit guidance. This would be Self-Supervised Learning in which the AI would be able to create its labels or predictions based on input data, generating a form of supervision. This system would be able to do facial detection and compare the Raw info with the DataSet. The system would also be able to make the most suitable and accurate decisions. With time this machine would be able to upgrade its memory (taking a human baby as an example/consideration). In a nutshell, our research aims to make AI systems not just powerful, but also easy to understand and trustworthy for everyone and make them Self-Supervised

Beyond Supervised: The Rise of Self-Supervised Learning in Autonomous Systems

Beyond Supervised: The Rise of Self-Supervised Learning in Autonomous Systems

Automated quantification of SARS-CoV-2 pneumonia with large vision model knowledge adaptation

BackgroundLarge vision models (LVM) pretrained by large datasets have demonstrated their enormous capacity to understand visual patterns and capture semantic information from images. We proposed a novel method of knowledge domain adaptation with pretrained LVM for a low-cost artificial intelligence (AI) model to quantify the severity of SARS-CoV-2 pneumonia based on frontal chest X-ray (CXR) images. MethodsOur method used the pretrained LVMs as the primary feature extractor and self-supervised contrastive learning for domain adaptation. An encoder with a 2048-dimensional feature vector output was first trained by self-supervised learning for knowledge domain adaptation. Then a multi-layer perceptron (MLP) was trained for the final severity prediction. A dataset with 2599 CXR images was used for model training and evaluation. ResultsThe model based on the pretrained vision transformer (ViT) and self-supervised learning achieved the best performance in cross validation, with mean squared error (MSE) of 23.83 (95 % CI 22.67–25.00) and mean absolute error (MAE) of 3.64 (95 % CI 3.54–3.73). Its prediction correlation has the R2 of 0.81 (95 % CI 0.79–0.82) and Spearman ρ of 0.80 (95 % CI 0.77–0.81), which are comparable to the current state-of-the-art (SOTA) methods trained by much larger CXR datasets. ConclusionThe proposed new method has achieved the SOTA performance to quantify the severity of SARS-CoV-2 pneumonia at a significantly lower cost. The method can be extended to other infectious disease detection or quantification to expedite the application of AI in medical research.

AI-enhanced predictive maintenance in hybrid roll-to-roll manufacturing integrating multi-sensor data and self-supervised learning

AI-enhanced predictive maintenance in hybrid roll-to-roll manufacturing integrating multi-sensor data and self-supervised learning

Enhancing Sound Source Localization via False Negative Elimination.

Sound source localization aims to localize objects emitting the sound in visual scenes. Recent works obtaining impressive results typically rely on contrastive learning. However, the common practice of randomly sampling negatives in prior arts can lead to the false negative issue, where the sounds semantically similar to visual instance are sampled as negatives and incorrectly pushed away from the visual anchor/query. As a result, this misalignment of audio and visual features could yield inferior performance. To address this issue, we propose a novel audio-visual learning framework which is instantiated with two individual learning schemes: self-supervised predictive learning (SSPL) and semantic-aware contrastive learning (SACL). SSPL explores image-audio positive pairs alone to discover semantically coherent similarities between audio and visual features, while a predictive coding module for feature alignment is introduced to facilitate the positive-only learning. In this regard SSPL acts as a negative-free method to eliminate false negatives. By contrast, SACL is designed to compact visual features and remove false negatives, providing reliable visual anchor and audio negatives for contrast. Different from SSPL, SACL releases the potential of audio-visual contrastive learning, offering an effective alternative to achieve the same goal. Comprehensive experiments demonstrate the superiority of our approach over the state-of-the-arts. Furthermore, we highlight the versatility of the learned representation by extending the approach to audio-visual event classification and object detection tasks. Code and models are available at: https://github.com/zjsong/SACL.

GMAEEG: A Self-Supervised Graph Masked Autoencoder for EEG Representation Learning.

Annotated electroencephalogram (EEG) data is the prerequisite for artificial intelligence-driven EEG autoanalysis. However, the scarcity of annotated data due to its high-cost and the resulted insufficient training limits the development of EEG autoanalysis. Generative self-supervised learning, represented by masked autoencoder, offers potential but struggles with non-Euclidean structures. To alleviate these challenges, this work proposes a self-supervised graph masked autoencoder for EEG representation learning, named GMAEEG. Concretely, a pretrained model is enriched with temporal and spatial representations through a masked signal reconstruction pretext task. A learnable dynamic adjacency matrix, initialized with prior knowledge, adapts to brain characteristics. Downstream tasks are achieved by finetuning pretrained parameters, with the adjacency matrix transferred based on task functional similarity. Experimental results demonstrate that with emotion recognition as the pretext task, GMAEEG reaches superior performance on various downstream tasks, including emotion, major depressive disorder, Parkinson's disease, and pain recognition. This study is the first to tailor the masked autoencoder specifically for EEG representation learning considering its non-Euclidean characteristics. Further, graph connection analysis based on GMAEEG may provide insights for future clinical studies.

Contrastive Machine Learning with Gamma Spectroscopy Data Augmentations for Detecting Shielded Radiological Material Transfers

Data analysis techniques can be powerful tools for rapidly analyzing data and extracting information that can be used in a latent space for categorizing observations between classes of data. Machine learning models that exploit learned data relationships can address a variety of nuclear nonproliferation challenges like the detection and tracking of shielded radiological material transfers. The high resource cost of manually labeling radiation spectra is a hindrance to the rapid analysis of data collected from persistent monitoring and to the adoption of supervised machine learning methods that require large volumes of curated training data. Instead, contrastive self-supervised learning on unlabeled spectra can enhance models that are built on limited labeled radiation datasets. This work demonstrates that contrastive machine learning is an effective technique for leveraging unlabeled data in detecting and characterizing nuclear material transfers demonstrated on radiation measurements collected at an Oak Ridge National Laboratory testbed, where sodium iodide detectors measure gamma radiation emitted by material transfers between the High Flux Isotope Reactor and the Radiochemical Engineering Development Center. Label-invariant data augmentations tailored for gamma radiation detection physics are used on unlabeled spectra to contrastively train an encoder, learning a complex, embedded state space with self-supervision. A linear classifier is then trained on a limited set of labeled data to distinguish transfer spectra between byproducts and tracked nuclear material using representations from the contrastively trained encoder. The optimized hyperparameter model achieves a balanced accuracy score of 80.30%. Any given model—that is, a trained encoder and classifier—shows preferential treatment for specific subclasses of transfer types. Regardless of the classifier complexity, a supervised classifier using contrastively trained representations achieves higher accuracy than using spectra when trained and tested on limited labeled data.

Multimodal representations of biomedical knowledge from limited training whole slide images and reports using deep learning

The increasing availability of biomedical data creates valuable resources for developing new deep learning algorithms to support experts, especially in domains where collecting large volumes of annotated data is not trivial. Biomedical data include several modalities containing complementary information, such as medical images and reports: images are often large and encode low-level information, while reports include a summarized high-level description of the findings identified within data and often only concerning a small part of the image. However, only a few methods allow to effectively link the visual content of images with the textual content of reports, preventing medical specialists from properly benefitting from the recent opportunities offered by deep learning models. This paper introduces a multimodal architecture creating a robust biomedical data representation encoding fine-grained text representations within image embeddings. The architecture aims to tackle data scarcity (combining supervised and self-supervised learning) and to create multimodal biomedical ontologies. The architecture is trained on over 6,000 colon whole slide Images (WSI), paired with the corresponding report, collected from two digital pathology workflows. The evaluation of the multimodal architecture involves three tasks: WSI classification (on data from pathology workflow and from public repositories), multimodal data retrieval, and linking between textual and visual concepts. Noticeably, the latter two tasks are available by architectural design without further training, showing that the multimodal architecture that can be adopted as a backbone to solve peculiar tasks. The multimodal data representation outperforms the unimodal one on the classification of colon WSIs and allows to halve the data needed to reach accurate performance, reducing the computational power required and thus the carbon footprint. The combination of images and reports exploiting self-supervised algorithms allows to mine databases without needing new annotations provided by experts, extracting new information. In particular, the multimodal visual ontology, linking semantic concepts to images, may pave the way to advancements in medicine and biomedical analysis domains, not limited to histopathology.

A robust operators’ cognitive workload recognition method based on denoising masked autoencoder

Identifying the cognitive workload of operators is crucial in complex human-automation collaboration systems. An excessive workload can lead to fatigue or accidents, while an insufficient workload may diminish situational awareness and efficiency. However, existing supervised learning-based methods for workload recognition are ineffective when dealing with imperfect input data, such as missing or noisy data, which is not practical in real applications. This study introduces a robust Electroencephalogram (EEG)-enabled cognitive workload recognition model using self-supervised learning. The proposed method, DMAEEG, combines the training strategies of denoising autoencoders and masked autoencoders, demonstrating strong robustness against noisy and incomplete data. More specifically, we adopt the temporal convolutional network and multi-head self-attention mechanisms as the backbone, effectively capturing both the temporal and spatial features from EEG. Extensive experiments are conducted to verify the effectiveness and robustness of the proposed method on an open dataset and a self-collected dataset. The results indicate that DMAEEG performs superior to other state-of-the-art across various evaluation metrics. Moreover, DMAEEG maintains high accuracy in workload inference even when EEG signals are corrupted with a high masking ratio or strong noises. This signifies its superiority in capturing robust intrinsic patterns from imperfect EEG data. The proposed method significantly contributes to decoding EEG signals for workload recognition in real-world applications, thereby enhancing the safety and reliability of human-automation interactions.

A Review of Deep Learning Based Video Action Recognition Techniques

Video action recognition is an important research direction in the field of computer vision and pattern recognition, with extensive applications in intelligent video surveillance, human-computer interaction, and sports analysis. The development of data storage and computing hardware over the past decade has driven a shift from traditional feature extraction and machine learning algorithms to deep learning-based approaches. This paper reviews the current state of development, problems, and future research directions of video action recognition techniques. Traditional methods are gradually being replaced by deep learning methods such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long-short-term memory networks (LSTMs). These methods automatically extract features and handle time-dependency, significantly improving the accuracy and robustness of action recognition. In particular, models based on the attention mechanism further enhance action recognition performance by dynamically adjusting the focus of attention, a current hot spot in research. Despite many advances, video action recognition still faces several challenges, including high computational resource requirements, complex model training, dataset bias issues, and variations in real-world application scenarios such as viewpoint changes, lighting changes, and occlusion. Future research can explore multi-modal fusion, lightweight models, self-supervised learning, and cross-domain transfer learning to improve the accuracy, robustness, and generalization of action recognition. The review provided aims to offer researchers a comprehensive perspective on the current state of development and future research directions of video action recognition technology.

Weakly-supervised learning-based pathology detection and localization in 3D chest CT scans.

Recent advancements in anomaly detection have paved the way for novel radiological reading assistance tools that support the identification of findings, aimed at saving time. The clinical adoption of such applications requires a low rate of false positives while maintaining high sensitivity. In light of recent interest and development in multi pathology identification, we present a novel method, based on a recent contrastive self-supervised approach, for multiple chest-related abnormality identification including low lung density area ("LLDA"), consolidation ("CONS"), nodules ("NOD") and interstitial pattern ("IP"). Our approach alerts radiologists about abnormal regions within a computed tomography (CT) scan by providing 3Dlocalization. We introduce a new method for the classification and localization of multiple chest pathologies in 3D Chest CT scans. Our goal is to distinguish four common chest-related abnormalities: "LLDA", "CONS", "NOD", "IP" and "NORMAL". This method is based on a 3D patch-based classifier with a Resnet backbone encoder pretrained leveraging recent contrastive self supervised approach and a fine-tuned classification head. We leverage the SimCLR contrastive framework for pretraining on an unannotated dataset of randomly selected patches and we then fine-tune it on a labeled dataset. During inference, this classifier generates probability maps for each abnormality across the CT volume, which are aggregated to produce a multi-label patient-level prediction. We compare different training strategies, including random initialization, ImageNet weight initialization, frozen SimCLR pretrained weights and fine-tuned SimCLR pretrained weights. Each training strategy is evaluated on a validation set for hyperparameter selection and tested on a test set. Additionally, we explore the fine-tuned SimCLR pretrained classifier for 3D pathology localization and conduct qualitativeevaluation. Validated on 111 chest scans for hyperparameter selection and subsequently tested on 251 chest scans with multi-abnormalities, our method achieves an AUROC of 0.931 (95% confidence interval [CI]: [0.9034, 0.9557], -value < 0.001) and 0.963 (95% CI: [0.952, 0.976], -value < 0.001) in the multi-label and binary (i.e., normal versus abnormal) settings, respectively. Notably, our method surpasses the area under the receiver operating characteristic (AUROC) threshold of 0.9 for two abnormalities: IP (0.974) and LLDA (0.952), while achieving values of 0.853 and 0.791 for NOD and CONS, respectively. Furthermore, our results highlight the superiority of incorporating contrastive pretraining within the patch classifier, outperforming Imagenet pretraining weights and non-pretrained counterparts with uninitialized weights (F1 score = 0.943, 0.792, and 0.677 respectively). Qualitatively, the method achieved a satisfactory 88.8% completeness rate in localization and maintained an 88.3% accuracy rate against falsepositives. The proposed method integrates self-supervised learning algorithms for pretraining, utilizes a patch-based approach for 3D pathology localization and develops an aggregation method for multi-label prediction at patient-level. It shows promise in efficiently detecting and localizing multiple anomalies within a singlescan.

Investigating Contrastive Pair Learning's Frontiers in Supervised, Semisupervised, and Self-Supervised Learning.

In recent years, contrastive learning has been a highly favored method for self-supervised representation learning, which significantly improves the unsupervised training of deep image models. Self-supervised learning is a subset of unsupervised learning in which the learning process is supervised by creating pseudolabels from the data themselves. Using supervised final adjustments after unsupervised pretraining is one way to take the most valuable information from a vast collection of unlabeled data and teach from a small number of labeled instances. This study aims firstly to compare contrastive learning with other traditional learning models; secondly to demonstrate by experimental studies the superiority of contrastive learning during classification; thirdly to fine-tune performance using pretrained models and appropriate hyperparameter selection; and finally to address the challenge of using contrastive learning techniques to produce data representations with semantic meaning that are independent of irrelevant factors like position, lighting, and background. Relying on contrastive techniques, the model efficiently captures meaningful representations by discerning similarities and differences between modified copies of the same image. The proposed strategy, involving unsupervised pretraining followed by supervised fine-tuning, improves the robustness, accuracy, and knowledge extraction of deep image models. The results show that even with a modest 5% of data labeled, the semisupervised model achieves an accuracy of 57.72%. However, the use of supervised learning with a contrastive approach and careful hyperparameter tuning increases accuracy to 85.43%. Further adjustment of the hyperparameters resulted in an excellent accuracy of 88.70%.

Foundation models in gastrointestinal endoscopic AI: Impact of architecture, pre-training approach and data efficiency

Pre-training deep learning models with large data sets of natural images, such as ImageNet, has become the standard for endoscopic image analysis. This approach is generally superior to training from scratch, due to the scarcity of high-quality medical imagery and labels. However, it is still unknown whether the learned features on natural imagery provide an optimal starting point for the downstream medical endoscopic imaging tasks. Intuitively, pre-training with imagery closer to the target domain could lead to better-suited feature representations. This study evaluates whether leveraging in-domain pre-training in gastrointestinal endoscopic image analysis has potential benefits compared to pre-training on natural images.To this end, we present a dataset comprising of 5,014,174 gastrointestinal endoscopic images from eight different medical centers (GastroNet-5M), and exploit self-supervised learning with SimCLRv2, MoCov2 and DINO to learn relevant features for in-domain downstream tasks. The learned features are compared to features learned on natural images derived with multiple methods, and variable amounts of data and/or labels (e.g. Billion-scale semi-weakly supervised learning and supervised learning on ImageNet-21k). The effects of the evaluation is performed on five downstream data sets, particularly designed for a variety of gastrointestinal tasks, for example, GIANA for angiodyplsia detection and Kvasir-SEG for polyp segmentation.The findings indicate that self-supervised domain-specific pre-training, specifically using the DINO framework, results into better performing models compared to any supervised pre-training on natural images. On the ResNet50 and Vision-Transformer-small architectures, utilizing self-supervised in-domain pre-training with DINO leads to an average performance boost of 1.63% and 4.62%, respectively, on the downstream datasets. This improvement is measured against the best performance achieved through pre-training on natural images within any of the evaluated frameworks.Moreover, the in-domain pre-trained models also exhibit increased robustness against distortion perturbations (noise, contrast, blur, etc.), where the in-domain pre-trained ResNet50 and Vision-Transformer-small with DINO achieved on average 1.28% and 3.55% higher on the performance metrics, compared to the best performance found for pre-trained models on natural images.Overall, this study highlights the importance of in-domain pre-training for improving the generic nature, scalability and performance of deep learning for medical image analysis. The GastroNet-5M pre-trained weights are made publicly available in our repository: huggingface.co/tgwboers/GastroNet-5M_Pretrained_Weights.