Unlabeled Images Research Articles

Learning to Detect Novel Species with SAM in the Wild

AbstractThis paper tackles the limitation of a closed-world object detection model that was trained on one species. The expectation for this model is that it will not generalize well to recognize the instances of new species if they were present in the incoming data stream. We propose a novel object detection framework for this open-world setting that is suitable for applications that monitor wildlife, ocean life, livestock, plant phenotype and crops that typically feature one species in the image. Our method leverages labeled samples from one species in combination with a novelty detection method and Segment Anything Model, a vision foundation model, to (1) identify the presence of new species in unlabeled images, (2) localize their instances, and (3) retrain the initial model with the localized novel class instances. The resulting integrated system assimilates and learns from unlabeled samples of the new classes while not “forgetting” the original species the model was trained on. We demonstrate our findings on two different domains, (1) wildlife detection and (2) plant detection. Our method achieves an AP of 56.2 (for 4 novel species) to 61.6 (for 1 novel species) for wildlife domain, without relying on any ground truth data in the background.

Effectiveness of Three Front-of-Pack Food Labels in Guiding Consumer Identification of Nutrients of Concern and Purchase Intentions in Kenya: A Randomized Controlled Trial

Background: Front-of-pack-labels (FOPLs) on packaged foods provide essential information to help consumers make informed dietary choices. However, evidence on their effectiveness, particularly in low- and middle-income countries like Kenya, is limited. Objective: This study assessed the effectiveness of three FOPLs in helping consumers identify nutrients of concern in packaged food products and influencing their purchase intention in Kenya. Methods: A total of 2198 shoppers from supermarkets in Nairobi, Mombasa, Kisumu, and Garissa were randomized into three groups: Red and Green Octagon label (RG), Red and Green Octagon with icons (RGI), and Black Octagon Warning label (WL). In the control phase, participants were shown unlabeled images of packaged foods, followed by questions. In the experimental phase, the same images were presented with one assigned FOPL, and participants responded again to the same set of questions. Differences in correct identification of nutrients of concern and changes in purchase intention were analyzed using frequency tables and Chi-Square tests, while modified Poisson regression assessed FOPL effectiveness. Results: FOPLs significantly improved correct nutrient identification and reduced the intention to purchase unhealthy foods, with the WL proving most effective. Conclusions: These findings highlight the potential of FOPLs, particularly the WL, as an effective regulatory tool for promoting healthier food choices in Kenya.

Developing a 10-Layer Retinal Segmentation for MacTel Using Semi-Supervised Learning.

Automated segmentation software in optical coherence tomography (OCT) devices is usually developed for and primarily tested on common diseases. Therefore segmentation accuracy of automated software can be limited in eyes with rare pathologies. We sought to develop a semisupervised deep learning segmentation model that segments 10 retinal layers and four retinal features in eyes with Macular Telangiectasia Type II (MacTel) using a small labeled dataset by leveraging unlabeled images. We compared our model against popular supervised and semisupervised models, as well as conducted ablation studies on the model itself. Our model significantly outperformed all other models in terms of intersection over union on the 10 retinal layers and two retinal features in the test dataset. For the remaining two features, the pre-retinal space above the internal limiting membrane and the background below the retinal pigment epithelium, all of the models performed similarly. Furthermore, we showed that using more unlabeled images improved the performance of our semisupervised model. Our model improves segmentation performance over supervised models by leveraging unlabeled data. This approach has the potential to improve segmentation performance for other diseases, where labeled data is limited but unlabeled data abundant. Improving automated segmentation of MacTel pathology on OCT imaging by leveraging unlabeled data may enable more accurate assessment of disease progression, and this approach may be useful for improving feature identification and location on OCT in other rare diseases as well.

Visual object detection model integrating deep active learning

The perception of surrounding objects by vehicle autonomous driving is an important means to ensure traffic safety. Object detection models based on deep learning are widely used, but they require a large amount of labeled data for training. This paper proposes an active visual object detection model that uses Gaussian mixture distribution to estimate the uncertainty of unlabeled images, reducing the dependence of model training on labeled data. First, a mixed density network is used as the detection head, and the image features extracted by the deep neural network are used as input to estimate the probability distribution of the classification and positioning of the target prediction box. Secondly, the classification score value of the target prediction box is mapped to the probability space, and the classification uncertainty of the target is calculated using the edge uncertainty; the positioning uncertainty of the target is measured using the prediction box positioning variance. Finally, the most unstable samples are selected for labeling. Compared with other typical active learning sampling strategies on the VOC dataset, the proposed method achieves the best performance, and the 54%data annotation amount can reach the performance of YOLOX supervised learning 98.8%, saving nearly 45% of the data annotation amount.

Dual structure-aware image filterings for semi-supervised medical image segmentation

Semi-supervised image segmentation has attracted great attention recently. The key is how to leverage unlabeled images in the training process. Most methods maintain consistent predictions of the unlabeled images under variations (e.g., adding noise/perturbations, or creating alternative versions) in the image and/or model level. In most image-level variation, medical images often have prior structure information, which has not been well explored. In this paper, we propose novel dual structure-aware image filterings (DSAIF) as the image-level variations for semi-supervised medical image segmentation. Motivated by connected filtering that simplifies image via filtering in structure-aware tree-based image representation, we resort to the dual contrast invariant Max-tree and Min-tree representation. Specifically, we propose a novel connected filtering that removes topologically equivalent nodes (i.e. connected components) having no siblings in the Max/Min-tree. This results in two filtered images preserving topologically critical structure. Applying the proposed DSAIF to mutually supervised networks decreases the consensus of their erroneous predictions on unlabeled images. This helps to alleviate the confirmation bias issue of overfitting to noisy pseudo labels of unlabeled images, and thus effectively improves the segmentation performance. Extensive experimental results on three benchmark datasets demonstrate that the proposed method significantly/consistently outperforms some state-of-the-art methods. The source codes will be publicly available.

EntroCap: Zero-shot image captioning with entropy-based retrieval

State-of-the-art image captioning often relies on supervised models trained on domain-specific image–text pairs, which can be expensive and time-consuming to annotate. Additionally, these models struggle with performance drops when applied to data from different distributions. To overcome these challenges, zero-shot image captioning has emerged, utilizing models trained on unlabeled image and text data. Pre-trained vision-language and generative language models have shown promise in this area, but existing methods do not fully leverage the cross-modal representation capabilities of large-scale models and tend to overlook smaller objects in images. We propose EntroCap, a novel zero-shot image captioning method that incorporates an Entropy-based Retrieval Strategy. During training, our Hierarchical Projector optimizes CLIP’s joint embeddings to capture comprehensive global signals with multi-level representations. During inference, the Entropy-based Retrieval Strategy adjusts prediction logits to better include smaller, less frequent objects. Additionally, our Balancing Gate mechanism guides the language model to produce more detailed and contextually relevant captions by balancing local and global signals. Extensive experiments on MSCOCO, Flickr30k, and NoCaps demonstrate that EntroCap outperforms previous zero-shot approaches in both in-domain and cross-domain settings. We also conduct experiments to quantitatively validate EntroCap’s effectiveness in describing small objects.

BI‐LAVA: Biocuration With Hierarchical Image Labelling Through Active Learning and Visual Analytics

AbstractIn the biomedical domain, taxonomies organize the acquisition modalities of scientific images in hierarchical structures. Such taxonomies leverage large sets of correct image labels and provide essential information about the importance of a scientific publication, which could then be used in biocuration tasks. However, the hierarchical nature of the labels, the overhead of processing images, the absence or incompleteness of labelled data and the expertise required to label this type of data impede the creation of useful datasets for biocuration. From a multi‐year collaboration with biocurators and text‐mining researchers, we derive an iterative visual analytics and active learning (AL) strategy to address these challenges. We implement this strategy in a system called BI‐LAVA—Biocuration with Hierarchical Image Labelling through Active Learning and Visual Analytics. BI‐LAVA leverages a small set of image labels, a hierarchical set of image classifiers and AL to help model builders deal with incomplete ground‐truth labels, target a hierarchical taxonomy of image modalities and classify a large pool of unlabelled images. BI‐LAVA's front end uses custom encodings to represent data distributions, taxonomies, image projections and neighbourhoods of image thumbnails, which help model builders explore an unfamiliar image dataset and taxonomy and correct and generate labels. An evaluation with machine learning practitioners shows that our mixed human–machine approach successfully supports domain experts in understanding the characteristics of classes within the taxonomy, as well as validating and improving data quality in labelled and unlabelled collections.

Uncertainty guided semi-supervised few-shot segmentation with prototype level fusion

Few-Shot Semantic Segmentation (FSS) aims to tackle the challenge of segmenting novel categories with limited annotated data. However, given the diversity among support-query pairs, transferring meta-knowledge to unseen categories poses a significant challenge, particularly in scenarios featuring substantial intra-class variance within an episode task. To alleviate this issue, we propose the Uncertainty Guided Adaptive Prototype Network (UGAPNet) for semi-supervised few-shot semantic segmentation. The key innovation lies in the generation of reliable pseudo-prototypes as an additional supplement to alleviate intra-class semantic bias. Specifically, we employ a shared meta-learner to produce segmentation results for unlabeled images in the pseudo-label prediction module. Subsequently, we incorporate an uncertainty estimation module to quantify the difference between prototypes extracted from query and support images, facilitating pseudo-label denoising. Utilizing these refined pseudo-label samples, we introduce a prototype rectification module to obtain effective pseudo-prototypes and generate a generalized adaptive prototype for the segmentation of query images. Furthermore, generalized few-shot semantic segmentation extends the paradigm of few-shot semantic segmentation by simultaneously segmenting both unseen and seen classes during evaluation. To address the challenge of confusion region prediction between these two categories, we further propose a novel Prototype-Level Fusion Strategy in the prototypical contrastive space. Extensive experiments conducted on two benchmarks demonstrate the effectiveness of the proposed UGAPNet and prototype-level fusion strategy. Our source code will be available on https://github.com/WHL182/UGAPNet.

Efficient semi-supervised hierarchical training for segmenting choroidal vessels and other structures on OCT images of multiple sclerosis patients

Optical coherence tomography (OCT) is a non-invasive imaging technique used to diagnose ocular and systemic diseases. Recently, several clinical studies have linked changes in different ocular layers to the development of multiple sclerosis (MS), so accurate segmentation of these structures has become an essential task. Unfortunately, segmenting the entire set of structures involved is a very difficult task, due to their large number and variability. These two factors hinder the labelling of images and therefore severely restrict the ability to achieve a large dataset with all structures manually annotated, limiting the use of a standard supervised approach. In this paper, we propose a semi-supervised learning methodology to robustly segment ocular structures in OCT images using a limited number of partially labeled images. Our methodology maximizes the information we can extract from labeled images through hierarchical learning, where multiple decoders are used to extract segmented structures progressively. We use a reconstruction loss function to provide structural coherence to the segmentation and a teacher-student strategy to effectively leverage the information present in the set of unlabeled images. In addition to the segmentation of labelled structures, this hierarchical approach allows segmenting structures that are not labelled in the dataset such as the choroidal vessels. To validate the proposed methodology, we have carried out extensive experimentation using two datasets with different characteristics. These experiments have demonstrated a great potential of this methodology to train networks efficiently with partially labeled images, which allows to accurately extract the main biomarkers linked to the development of MS.

Improved Road Extraction Models through Semi-Supervised Learning with ACCT

Improving the performance and reducing the training cost of road extraction models in the absence of samples is important for updating road maps. Despite the success of recent road extraction models on standard datasets, they often fail to perform when applied to new datasets or real-world scenarios where labeled samples are not available. In this paper, our focus diverges from the typical quest to pinpoint the optimal road extraction model or evaluate generalization prowess across models. Instead, we propose a method called Asymmetric Consistent Co-Training (ACCT) to train existing road extraction models faster and make them perform better in new scenarios lacking samples. ACCT uses two models with different structures and a supervision module to enhance accuracy through mutual learning. Labeled and unlabeled images are processed by both models to generate road maps from different perspectives. The supervision module ensures consistency between predictions by computing losses based on labeling status. ACCT iteratively adjusts parameters using unlabeled data, improving generalization. Empirical evaluations show that ACCT improves IoU by 2.79% to 10.26% using only 1/8 of the labeled data compared to fully supervised methods. It also reduces parameters by over 49% compared to state-of-the-art semi-supervised methods while maintaining similar accuracy. These results highlight the potential of leveraging large amounts of unlabeled data to enhance road extraction models as data acquisition technology advances.

Semi-Supervised Interior Decoration Style Classification with Contrastive Mutual Learning

Precisely identifying interior decoration styles holds substantial significance in directing interior decoration practices. Nevertheless, constructing accurate models for the automatic classification of interior decoration styles remains challenging due to the scarcity of expert annotations. To address this problem, we propose a novel pseudo-label-guided contrastive mutual learning framework (PCML) for semi-supervised interior decoration style classification by harnessing large amounts of unlabeled data. Specifically, PCML introduces two distinct subnetworks and selectively utilizes the diversified pseudo-labels generated by each for mutual supervision, thereby mitigating the issue of confirmation bias. For labeled images, the inconsistent pseudo-labels generated by the two subnetworks are employed to identify images that are prone to misclassification. We then devise an inconsistency-aware relearning (ICR) regularization model to perform a review training process. For unlabeled images, we introduce a class-aware contrastive learning (CCL) regularization to learn their discriminative feature representations using the corresponding pseudo-labels. Since the use of distinct subnetworks reduces the risk of both models producing identical erroneous pseudo-labels, CCL can reduce the possibility of noise data sampling to enhance the effectiveness of contrastive learning. The performance of PCML is evaluated on five interior decoration style image datasets. For the average AUC, accuracy, sensitivity, specificity, precision, and F1 scores, PCML obtains improvements of 1.67%, 1.72%, 3.65%, 1.0%, 4.61%, and 4.66% in comparison with the state-of-the-art method, demonstrating the superiority of our method.

A Novel Perturbation Consistency Framework in Semi-Supervised Medical Image Segmentation

Semi-supervised medical image segmentation models often face challenges such as empirical mismatch and data imbalance. Traditional methods, like the two-stream perturbation model, tend to over-rely on strong perturbation, leaving weak perturbation and labeled images underutilized. To overcome these challenges, we propose an innovative hybrid copy-paste (HCP) method within the strong perturbation branch, encouraging unlabeled images to learn more comprehensive semantic information from labeled images and narrowing the empirical distribution gap. Additionally, we integrate contrastive learning into the weak perturbation branch, where contrastive learning samples are selected through semantic grouping contrastive sampling (SGCS) to address memory and variance issues. This sampling strategy ensures more effective use of weak perturbation data. This approach is particularly advantageous for pixel segmentation tasks with severely limited labels. Finally, our approach is validated on the public ACDC (Automated Cardiac Diagnosis Challenge) dataset, achieving a 90.6% DICE score, with just 7% labeled data. These results demonstrate the effectiveness of our method in improving segmentation performance with limited labeled data.

Detecting floating litter in freshwater bodies with semi-supervised deep learning

Researchers and practitioners have extensively utilized supervised Deep Learning methods to quantify floating litter in rivers and canals. These methods require the availability of large amount of labeled data for training. The labeling work is expensive and laborious, resulting in small open datasets available in the field compared to the comprehensive datasets for computer vision, e.g., ImageNet. Fine-tuning models pre-trained on these larger datasets helps improve litter detection performances and reduces data requirements. Yet, the effectiveness of using features learned from generic datasets is limited in large-scale monitoring, where automated detection must adapt across different locations, environmental conditions, and sensor settings. To address this issue, we propose a two-stage semi-supervised learning method to detect floating litter based on the Swapping Assignments between multiple Views of the same image (SwAV). SwAV is a self-supervised learning approach that learns the underlying feature representation from unlabeled data. In the first stage, we used SwAV to pre-train a ResNet50 backbone architecture on about 100k unlabeled images. In the second stage, we added new layers to the pre-trained ResNet50 to create a Faster R-CNN architecture, and fine-tuned it with a limited number of labeled images (≈1.8k images with 2.6k annotated litter items). We developed and validated our semi-supervised floating litter detection methodology for images collected in canals and waterways of Delft (the Netherlands) and Jakarta (Indonesia). We tested for out-of-domain generalization performances in a zero-shot fashion using additional data from Ho Chi Minh City (Vietnam), Amsterdam and Groningen (the Netherlands). We benchmarked our results against the same Faster R-CNN architecture trained via supervised learning alone by fine-tuning ImageNet pre-trained weights. The findings indicate that the semi-supervised learning method matches or surpasses the supervised learning benchmark when tested on new images from the same training locations. We measured better performances when little data (≈200 images with about 300 annotated litter items) is available for fine-tuning and with respect to reducing false positive predictions. More importantly, the proposed approach demonstrates clear superiority for generalization on the unseen locations, with improvements in average precision of up to 12.7%. We attribute this superior performance to the more effective high-level feature extraction from SwAV pre-training from relevant unlabeled images. Our findings highlight a promising direction to leverage semi-supervised learning for developing foundational models, which have revolutionized artificial intelligence applications in most fields. By scaling our proposed approach with more data and compute, we can make significant strides in monitoring to address the global challenge of litter pollution in water bodies.

Layer-Wise Mutual Information Meta-Learning Network for Few-Shot Segmentation.

The goal of few-shot segmentation (FSS) is to segment unlabeled images belonging to previously unseen classes using only a limited number of labeled images. The main objective is to transfer label information effectively from support images to query images. In this study, we introduce a novel meta-learning framework called layer-wise mutual information (LayerMI), which enhances the propagation of label information by maximizing the mutual information (MI) between support and query features at each layer. Our approach involves the utilization of a LayerMI Block based on information-theoretic co-clustering. This block performs online co-clustering on the joint probability distribution obtained from each layer, generating a target-specific attention map. The LayerMI Block can be seamlessly integrated into the meta-learning framework and applied to all convolutional neural network (CNN) layers without altering the training objectives. Notably, the LayerMI Block not only maximizes MI between support and query features but also facilitates internal clustering within the image. Extensive experiments demonstrate that LayerMI significantly enhances the performance of baseline and achieves competitive performance compared to state-of-the-art methods on three challenging benchmarks: PASCAL-5 i , COCO-20 i , and FSS-1000.

A Semi-Supervised Transformer-Based Deep Learning Framework for Automated Tooth Segmentation and Identification on Panoramic Radiographs.

Automated tooth segmentation and identification on dental radiographs are crucial steps in establishing digital dental workflows. While deep learning networks have been developed for these tasks, their performance has been inferior in partially edentulous individuals. This study proposes a novel semi-supervised Transformer-based framework (SemiTNet), specifically designed to improve tooth segmentation and identification performance on panoramic radiographs, particularly in partially edentulous cases, and establish an open-source dataset to serve as a unified benchmark. A total of 16,317 panoramic radiographs (1589 labeled and 14,728 unlabeled images) were collected from various datasets to create a large-scale dataset (TSI15k). The labeled images were divided into training and test sets at a 7:1 ratio, while the unlabeled images were used for semi-supervised learning. The SemiTNet was developed using a semi-supervised learning method with a label-guided teacher-student knowledge distillation strategy, incorporating a Transformer-based architecture. The performance of SemiTNet was evaluated on the test set using the intersection over union (IoU), Dice coefficient, precision, recall, and F1 score, and compared with five state-of-the-art networks. Paired t-tests were performed to compare the evaluation metrics between SemiTNet and the other networks. SemiTNet outperformed other networks, achieving the highest accuracy for tooth segmentation and identification, while requiring minimal model size. SemiTNet's performance was near-perfect for fully dentate individuals (all metrics over 99.69%) and excellent for partially edentulous individuals (all metrics over 93%). In edentulous cases, SemiTNet obtained statistically significantly higher tooth identification performance than all other networks. The proposed SemiTNet outperformed previous high-complexity, state-of-the-art networks, particularly in partially edentulous cases. The established open-source TSI15k dataset could serve as a unified benchmark for future studies.

ECCT: Efficient Contrastive Clustering via Pseudo-Siamese Vision Transformer and Multi-view Augmentation

Image clustering aims to divide a set of unlabeled images into multiple clusters. Recently, clustering methods based on contrastive learning have attracted much attention due to their ability to learn discriminative feature representations. Nevertheless, existing clustering algorithms face challenges in capturing global information and preserving semantic continuity. Additionally, these methods often exhibit relatively singular feature distributions, limiting the full potential of contrastive learning in clustering. These problems can have a negative impact on the performance of image clustering. To address the above problems, we propose a deep clustering framework termed Efficient Contrastive Clustering via Pseudo-Siamese Vision Transformer and Multi-view Augmentation (ECCT). The core idea is to introduce Vision Transformer (ViT) to provide the global view, and improve it with Hilbert Patch Embedding (HPE) module to construct a new ViT branch. Finally, we fuse the features extracted from the two ViT branches to obtain both global view and semantic coherence. In addition, we employ multi-view random aggressive augmentation to broaden the feature distribution, enabling the model to learn more comprehensive and richer contrastive features. Our results on five datasets demonstrate that ECCT outperforms previous clustering methods. In particular, the ARI metric of ECCT on the STL-10 (ImageNet-Dogs) dataset is 0.852 (0.424), which is 10.3% (4.8%) higher than the best baseline.

FPL+: Filtered Pseudo Label-Based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation.

Adapting a medical image segmentation model to a new domain is important for improving its cross-domain transferability, and due to the expensive annotation process, Unsupervised Domain Adaptation (UDA) is appealing where only unlabeled images are needed for the adaptation. Existing UDA methods are mainly based on image or feature alignment with adversarial training for regularization, and they are limited by insufficient supervision in the target domain. In this paper, we propose an enhanced Filtered Pseudo Label (FPL+)-based UDA method for 3D medical image segmentation. It first uses cross-domain data augmentation to translate labeled images in the source domain to a dual-domain training set consisting of a pseudo source-domain set and a pseudo target-domain set. To leverage the dual-domain augmented images to train a pseudo label generator, domain-specific batch normalization layers are used to deal with the domain shift while learning the domain-invariant structure features, generating high-quality pseudo labels for target-domain images. We then combine labeled source-domain images and target-domain images with pseudo labels to train a final segmentor, where image-level weighting based on uncertainty estimation and pixel-level weighting based on dual-domain consensus are proposed to mitigate the adverse effect of noisy pseudo labels. Experiments on three public multi-modal datasets for Vestibular Schwannoma, brain tumor and whole heart segmentation show that our method surpassed ten state-of-the-art UDA methods, and it even achieved better results than fully supervised learning in the target domain in some cases.

3D Vessel Segmentation With Limited Guidance of 2D Structure-Agnostic Vessel Annotations.

Delineating 3D blood vessels of various anatomical structures is essential for clinical diagnosis and treatment, however, is challenging due to complex structure variations and varied imaging conditions. Although recent supervised deep learning models have demonstrated their superior capacity in automatic 3D vessel segmentation, the reliance on expensive 3D manual annotations and limited capacity for annotation reuse among different vascular structures hinder their clinical applications. To avoid the repetitive and costly annotating process for each vascular structure and make full use of existing annotations, this paper proposes a novel 3D shape-guided local discrimination (3D-SLD) model for 3D vascular segmentation under limited guidance from public 2D vessel annotations. The primary hypothesis is that 3D vessels are composed of semantically similar voxels and often exhibit tree-shaped morphology. Accordingly, the 3D region discrimination loss is firstly proposed to learn the discriminative representation measuring voxel-wise similarities and cluster semantically consistent voxels to form the candidate 3D vascular segmentation in unlabeled images. Secondly, the shape distribution from existing 2D structure-agnostic vessel annotations is introduced to guide the 3D vessels with the tree-shaped morphology by the adversarial shape constraint loss. Thirdly, to enhance training stability and prediction credibility, the highlighting-reviewing-summarizing (HRS) mechanism is proposed. This mechanism involves summarizing historical models to maintain temporal consistency and identifying credible pseudo labels as reliable supervision signals. Only guided by public 2D coronary artery annotations, our method achieves results comparable to SOTA barely-supervised methods in 3D cerebrovascular segmentation, and the best DSC in 3D hepatic vessel segmentation, demonstrating the effectiveness of our method.

PolyMeme: Fine-Grained Internet Meme Sensing.

Internet memes are a special type of digital content that is shared through social media. They have recently emerged as a popular new format of media communication. They are often multimodal, combining text with images and aim to express humor, irony, sarcasm, or sometimes convey hatred and misinformation. Automatically detecting memes is important since it enables tracking of social and cultural trends and issues related to the spread of harmful content. While memes can take various forms and belong to different categories, such as image macros, memes with labeled objects, screenshots, memes with text out of the image, and funny images, existing datasets do not account for the diversity of meme formats, styles and content. To bridge this gap, we present the PolyMeme dataset, which comprises approximately 27 K memes from four categories. This was collected from Reddit and a part of it was manually labelled into these categories. Using the manual labels, deep learning networks were trained to classify the unlabelled images with an estimated error rate of 7.35%. The introduced meme dataset in combination with existing datasets of regular images were used to train deep learning networks (ResNet, ViT) on meme detection, exhibiting very high accuracy levels (98% on the test set). In addition, no significant gains were identified from the use of regular images containing text.

Masked autoencoder: influence of self-supervised pretraining on object segmentation in industrial images

The amount of labelled data in industrial use cases is limited because the annotation process is time-consuming and costly. As in research, self-supervised pretraining such as MAE resulted in training segmentation models with fewer labels, this is also an interesting direction for industry. The reduction of required labels is achieved with large amounts of unlabelled images for the pretraining that aims to learn image features. This paper analyses the influence of MAE pretraining on the efficiency of label usage for semantic segmentation with UNETR. This is investigated for the use case of log-yard cranes. Additionally, two transfer learning cases with respect to crane type and perspective are considered in the context of label-efficiency. The results show that MAE is successfully applicable to the use case. With respect to the segmentation, an IoU improvement of 3.26% is reached while using 2000 labels. The strongest positive influence is found for all experiments in the lower label amounts. The highest effect is achieved with transfer learning regarding cranes, where IoU and Recall increase about 4.31% and 8.58%, respectively. Further analyses show that improvements result from a better distinction between the background and the segmented crane objects.