Pixel-wise Annotations Research Articles

Real-time segmentation of biliary structure in pure laparoscopic donor hepatectomy

Pure laparoscopic donor hepatectomy (PLDH) has become a standard practice for living donor liver transplantation in expert centers. Accurate understanding of biliary structures is crucial during PLDH to minimize the risk of complications. This study aims to develop a deep learning-based segmentation model for real-time identification of biliary structures, assisting surgeons in determining the optimal transection site during PLDH. A single-institution retrospective feasibility analysis was conducted on 30 intraoperative videos of PLDH. All videos were selected for their use of the indocyanine green near-infrared fluorescence technique to identify biliary structure. From the analysis, 10 representative frames were extracted from each video specifically during the bile duct division phase, resulting in 300 frames. These frames underwent pixel-wise annotation to identify biliary structures and the transection site. A segmentation task was then performed using a DeepLabV3+ algorithm, equipped with a ResNet50 encoder, focusing on the bile duct (BD) and anterior wall (AW) for transection. The model’s performance was evaluated using the dice similarity coefficient (DSC). The model predicted biliary structures with a mean DSC of 0.728 ± 0.01 for BD and 0.429 ± 0.06 for AW. Inference was performed at a speed of 15.3 frames per second, demonstrating the feasibility of real-time recognition of anatomical structures during surgery. The deep learning-based semantic segmentation model exhibited promising performance in identifying biliary structures during PLDH. Future studies should focus on validating the clinical utility and generalizability of the model and comparing its efficacy with current gold standard practices to better evaluate its potential clinical applications.

Domain generalization for semantic segmentation: a survey

Deep neural networks (DNNs) have proven explicit contributions in making autonomous driving cars and related tasks such as semantic segmentation, motion tracking, object detection, sensor fusion, and planning. However, in challenging situations, DNNs are not generalizable because of the inherent domain shift due to the nature of training under the i.i.d. assumption. The goal of semantic segmentation is to preserve information from a given image into multiple meaningful categories for visual understanding. Particularly for semantic segmentation, pixel-wise annotation is extremely costly and not always feasible. Domain generalization for semantic segmentation aims to learn pixel-level semantic labels from multiple source domains and generalize to predict pixel-level semantic labels on multiple unseen target domains. In this survey, for the first time, we present a comprehensive review of DG for semantic segmentation. we present a comprehensive summary of recent works related to domain generalization in semantic segmentation, which establishes the importance of generalizing to new environments of segmentation models. Although domain adaptation has gained more attention in segmentation tasks than domain generalization, it is still worth unveiling new trends that are adopted from domain generalization methods in semantic segmentation. We cover most of the recent and dominant DG methods in the context of semantic segmentation and also provide some other related applications. We conclude this survey by highlighting the future directions in this area.

P-CNN: Percept-CNN for semantic segmentation

ABSTRACT The task of image segmentation remains a fundamental challenge, in the field of computer vision. Convolutional Neural Networks (CNNs) have achieved significant success in this field, yet there are some limitations in the conventional approach. The process of accurate, pixel-wise image annotation is time-consuming, as well as requires more human effort. These problems are addressed by the proposed method called as Percept-CNN (P-CNN), which extracts the power of percepts, pixels responsible for the higher activations at each level. From each layer, percepts are extracted during the forward propagation. These percepts are then passed onto the subsequent layers, enabling the model to focus only on the useful visual information. The proposed method with Percept Convolution can potentially eliminate the complex and time-consuming task of image annotation without affecting the segmentation accuracy. Since the model focuses only on the useful salient visual information, it tends to reduce the extraction of the redundant features, which doesn’t really contribute towards the final goal. This makes the model to be more robust, accurate and efficient. The proposed model was able to perform semantic segmentation without pixelwise annotations with an accuracy of 67% when tested on Oxford IIIT pet dataset.

Utilizing deep learning model for assessing melanocytic density in resection margins of lentigo maligna

BackgroundSurgical excision with clear histopathological margins is the preferred treatment to prevent progression of lentigo maligna (LM) to invasive melanoma. However, the assessment of resection margins on sun-damaged skin is challenging. We developed a deep learning model for detection of melanocytes in resection margins of LM.MethodsIn total, 353 whole slide images (WSIs) were included. 295 WSIs were used for training and 58 for validation and testing. The algorithm was trained with 3,973 manual pixel-wise annotations. The AI analyses were compared to those of three blinded dermatopathologists and two pathology residents, who performed their evaluations without AI and AI-assisted. Immunohistochemistry (SOX10) served as the reference standard. We used a dichotomized cutoff for low and high risk of recurrence (≤ 25 melanocytes in an area of 0.5 mm for low risk and > 25 for high risk).ResultsThe AI model achieved an area under the receiver operating characteristic curve (AUC) of 0.84 in discriminating margins with low and high recurrence risk. In comparison, the AUC for dermatopathologists ranged from 0.72 to 0.90 and for the residents in pathology, 0.68 to 0.80. Additionally, with aid of the AI model the performance of two pathologists significantly improved.ConclusionsThe deep learning showed notable accuracy in detecting resection margins of LM with a high versus low risk of recurrence. Furthermore, the use of AI improved the performance of 2/5 pathologists. This automated tool could aid pathologists in the assessment or pre-screening of LM margins.

Quantification of cardiac capillarization in basement-membrane-immunostained myocardial slices using Segment Anything Model

Decreased myocardial capillary density has been reported as an important histopathological feature associated with various heart disorders. Quantitative assessment of cardiac capillarization typically involves double immunostaining of cardiomyocytes (CMs) and capillaries in myocardial slices. In contrast, single immunostaining of basement membrane protein is a straightforward approach to simultaneously label CMs and capillaries, presenting fewer challenges in background staining. However, subsequent image analysis always requires expertise and laborious manual work to identify and segment CMs/capillaries. Here, we developed an image analysis tool, AutoQC, for automatic identification and segmentation of CMs and capillaries in immunofluorescence images of basement membrane. Commonly used capillarization-related measurements can be derived from segmentation results. By leveraging the power of a pre-trained segmentation model (Segment Anything Model, SAM) via prompt engineering, the training of AutoQC required only a small dataset with bounding box annotations instead of pixel-wise annotations. AutoQC outperformed SAM (without prompt engineering) and YOLOv8-Seg, a state-of-the-art instance segmentation model, in both instance segmentation and capillarization assessment. Thus, AutoQC, featuring a weakly supervised algorithm, enables automatic segmentation and high-throughput, high-accuracy capillarization assessment in basement-membrane-immunostained myocardial slices. This approach reduces the training workload and eliminates the need for manual image analysis once AutoQC is trained.

DiffuPrompter: Pixel-Level Automatic Annotation for High-Resolution Remote Sensing Images with Foundation Models

Instance segmentation is pivotal in remote sensing image (RSI) analysis, aiding in many downstream tasks. However, annotating images with pixel-wise annotations is time-consuming and laborious. Despite some progress in automatic annotation, the performance of existing methods still needs improvement due to the high precision requirements for pixel-level annotation and the complexity of RSIs. With the support of large-scale data, some foundational models have made significant progress in semantic understanding and generalization capabilities. In this paper, we delve deep into the potential of the foundational models in automatic annotation and propose a training-free automatic annotation method called DiffuPrompter, achieving pixel-level automatic annotation of RSIs. Extensive experimental results indicate that the proposed method can provide reliable pseudo-labels, significantly reducing the annotation costs of the segmentation task. Additionally, the cross-domain validation experiments confirm the powerful effectiveness of large-scale pseudo-data in improving model generalization performance.

Perturbed Progressive Learning for Semisupervised Defect Segmentation.

Recently, with the development of intelligent manufacturing, the demand for surface defect inspection has been increasing. Deep learning has achieved promising results in defect inspection. However, due to the rareness of defect data and the difficulties of pixelwise annotation, the existing supervised defect inspection methods are too inferior to be implemented in practice. To solve the problem of defect segmentation with few labeled data, we propose a simple and efficient method for semisupervised defect segmentation (SSDS), named perturbed progressive learning (PPL). On the one hand, PPL decouples the predictions of student and teacher networks as well as alleviates overfitting on noisy pseudo-labels. On the other hand, PPL encourages consistency across various perturbations in a broader stagewise scope, alleviating drift caused by the noisy pseudo-labels. Specifically, PPL contains two training stages. In the first stage, the teacher network gives the unlabeled data with pseudo-labels that are divided into the easy and hard groups. The labeled data and the unlabeled data in the easy group with their perturbation are both used to train for a better-performing student network. In the second stage, the unlabeled data in the hard group are predicted by the obtained student network, so the refined pseudo-labeled data are enlarged. All the pseudo-labeling data and labeled data with their perturbation are used to retrain the student network, progressively improving the defect feature representation. We build a mobile screen defect dataset (MSDD-3) with three classes of defects. PPL is implemented on MSDD-3 as well as other public datasets. Extensive experimental results demonstrate that PPL significantly surpasses the state-of-the-art methods across all evaluation partition protocols.

Random bridge generator as a platform for developing computer vision-based structural inspection algorithms

Recent advances in computer vision algorithms have transformed the bridge visual inspection process. Those algorithms typically require large amounts of annotated data, which is lacking for generic bridge inspection scenarios. To address this challenge efficiently, this research designs, develops, and demonstrates a platform that can provide synthetic datasets and testing environments, termed Random Bridge Generator (RBG). The RBG produces photo-realistic 3D synthetic environments of six types of bridges randomly, automatically, and procedurally. Following relevant standards and design practice, the RBG creates random cross-sectional shapes, converts those shapes into bridge components, and assembles the components into bridges. The effectiveness of the RBG is demonstrated by producing a dataset (RBG Dataset) containing 10,753 images with pixel-wise annotations, rendered in 250 different synthetic environments. Significant diversity of the photo-realistic bridge inspection environments has been achieved, while all structural components strictly conform to the definitions derived from structural engineering documents. The use of the RBG dataset has been demonstrated by training a deep semantic segmentation algorithm with 101 convolutional layers, showing successful segmentation results for both major and minor structural components. The developed RBG is expected to enhance the level of automation in bridge visual inspection process. The Python code for RBG is made public at: https://github.com/chenghaojia2323/Random-Bridge-Generator.git.

A publicly available deep learning model and dataset for segmentation of breast, fibroglandular tissue, and vessels in breast MRI

Breast density, or the amount of fibroglandular tissue (FGT) relative to the overall breast volume, increases the risk of developing breast cancer. Although previous studies have utilized deep learning to assess breast density, the limited public availability of data and quantitative tools hinders the development of better assessment tools. Our objective was to (1) create and share a large dataset of pixel-wise annotations according to well-defined criteria, and (2) develop, evaluate, and share an automated segmentation method for breast, FGT, and blood vessels using convolutional neural networks. We used the Duke Breast Cancer MRI dataset to randomly select 100 MRI studies and manually annotated the breast, FGT, and blood vessels for each study. Model performance was evaluated using the Dice similarity coefficient (DSC). The model achieved DSC values of 0.92 for breast, 0.86 for FGT, and 0.65 for blood vessels on the test set. The correlation between our model’s predicted breast density and the manually generated masks was 0.95. The correlation between the predicted breast density and qualitative radiologist assessment was 0.75. Our automated models can accurately segment breast, FGT, and blood vessels using pre-contrast breast MRI data. The data and the models were made publicly available.

Improving Semantic Segmentation via Efficient Self-Training.

Starting from the seminal work of Fully Convolutional Networks (FCN), there has been significant progress on semantic segmentation. However, deep learning models often require large amounts of pixelwise annotations to train accurate and robust models. Given the prohibitively expensive annotation cost of segmentation masks, we introduce a self-training framework in this paper to leverage pseudo labels generated from unlabeled data. In order to handle the data imbalance problem of semantic segmentation, we propose a centroid sampling strategy to uniformly select training samples from every class within each epoch. We also introduce a fast training schedule to alleviate the computational burden. This enables us to explore the usage of large amounts of pseudo labels. Our Centroid Sampling based Self-Training framework (CSST) achieves state-of-the-art results on Cityscapes and CamVid datasets. On PASCAL VOC 2012 test set, our models trained with the original train set even outperform the same models trained on the much bigger augmented train set. This indicates the effectiveness of CSST when there are fewer annotations. We also demonstrate promising few-shot generalization capability from Cityscapes to BDD100K and from Cityscapes to Mapillary datasets.

Weakly supervised learning for multi-class medical image segmentation via feature decomposition

Weakly supervised learning with image-level labels, releasing deep learning from highly labor-intensive pixel-wise annotation, has gained great attention for medical image segmentation. However, existing weakly supervised methods are mainly designed for single-class segmentation while leaving multi-class medical image segmentation rarely-explored. Different from natural images, label symbiosis, together with location adjacency, are much more common in medical images, making it more challenging for multi-class segmentation. In this paper, we propose a novel weakly supervised learning method for multi-class medical image segmentation with image-level labels. In terms of the multi-class classification backbone, a multi-level classification network encoding multi-scale features is proposed to produce binary predictions, together with the corresponding CAMs, of each class separately. To address the above issues (i.e., label symbiosis and location adjacency), a feature decomposition module based on semantic affinity is first proposed to learn both class-independent and class-dependent features by maximizing the inter-class feature distance. Through a cross-guidance loss to jointly utilize the above features, label symbiosis is largely alleviated. In terms of location adjacency, a mutually exclusive loss is constructed to minimize the overlap among regions corresponding to different classes. Experimental results on three datasets demonstrate the superior performance of the proposed weakly-supervised framework for both single-class and multi-class medical image segmentation. We believe the analysis in this paper would shed new light on future work for multi-class medical image segmentation. The source code of this paper is publicly available at https://github.com/HustAlexander/MCWSS.

STEdge: Self-Training Edge Detection With Multilayer Teaching and Regularization.

Learning-based edge detection has hereunto been strongly supervised with pixel-wise annotations which are tedious to obtain manually. We study the problem of self-training edge detection, leveraging the untapped wealth of large-scale unlabeled image datasets. We design a self-supervised framework with multilayer regularization and self-teaching. In particular, we impose a consistency regularization which enforces the outputs from each of the multiple layers to be consistent for the input image and its perturbed counterpart. We adopt L0-smoothing as the "perturbation" to encourage edge prediction lying on salient boundaries following the cluster assumption in self-supervised learning. Meanwhile, the network is trained with multilayer supervision by pseudo labels which are initialized with Canny edges and then iteratively refined by the network as the training proceeds. The regularization and self-teaching together attain a good balance of precision and recall, leading to a significant performance boost over supervised methods, with lightweight refinement on the target dataset. Through extensive experiments, our method demonstrates strong cross-dataset generality and can improve the original performance of edge detectors after self-training and fine-tuning.

Semi-Supervised Semantic Segmentation for Light Field Images Using Disparity Information.

Light field (LF) images enable numerous applications due to their ability to capture information for multiple views. Semantic segmentation is an essential task for LF scene understanding. However, existing supervised methods heavily rely on a large number of pixel-wise annotations. To relieve this problem, we propose a semi-supervised LF semantic segmentation method that requires only a small subset of labeled data and harnesses the LF disparity information. First, we design an unsupervised disparity estimation network, which can determine the disparity map for every view. With the estimated disparity maps, we generate pseudo-labels along with their weight maps for the peripheral views when only the labels of central views are available. We then merge the predictions from multiple views to obtain more reliable pseudo-labels for unlabeled data, and introduce a disparity-semantics consistency loss to enforce structure similarity. Moreover, we develop a comprehensive contrastive learning scheme that includes a pixel-level strategy to enhance feature representations and an object-level strategy to improve segmentation for individual objects. Our method demonstrates state-of-the-art performance on the benchmark LF semantic segmentation dataset under a variety of training settings and achieves comparable performance to supervised methods when trained under 1/2 protocol.

Transfer Learning with Pretrained Convolutional Neural Network for Automated Gleason Grading of Prostate Cancer Tissue Microarrays.

The Gleason grading system has been the most effective prediction for prostate cancer patients. This grading system provides this possibility to assess prostate cancer's aggressiveness and then constitutes an important factor for stratification and therapeutic decisions. However, determining Gleason grade requires highly-trained pathologists and is time-consuming and tedious, and suffers from inter-pathologist variability. To remedy these limitations, this paper introduces an automatic methodology based on transfer learning with pretrained convolutional neural networks (CNNs) for automatic Gleason grading of prostate cancer tissue microarray (TMA). Fifteen pretrained (CNNs): Efficient Nets (B0-B5), NasNetLarge, NasNetMobile, InceptionV3, ResNet-50, SeResnet 50, Xception, DenseNet121, ResNext50, and inception_resnet_v2 were fine-tuned on a dataset of prostate carcinoma TMA images. Six pathologists separately identified benign and cancerous areas for each prostate TMA image by allocating benign, 3, 4, or 5 Gleason grade for 244 patients. The dataset was labeled by these pathologists and majority vote was applied on pixel-wise annotations to obtain a unified label. Results showed the NasnetLarge architecture is the best model among them in the classification of prostate TMA images of 244 patients with accuracy of 0.93 and area under the curve of 0.98. Our study can act as a highly trained pathologist to categorize the prostate cancer stages with more objective and reproducible results.

Transformer-Based Weakly Supervised Learning for Whole Slide Lung Cancer Image Classification.

Image analysis can play an important role in supporting histopathological diagnoses of lung cancer, with deep learning methods already achieving remarkable results. However, due to the large scale of whole-slide images (WSIs), creating manual pixel-wise annotations from expert pathologists is expensive and time-consuming. In addition, the heterogeneity of tumors and similarities in the morphological phenotype of tumor subtypes have caused inter-observer variability in annotations, which limits optimal performance. Effective use of weak labels could potentially alleviate these issues. In this paper, we propose a two-stage transformer-based weakly supervised learning framework called Simple Shuffle-Remix Vision Transformer (SSRViT). Firstly, we introduce a Shuffle-Remix Vision Transformer (SRViT) to retrieve discriminative local tokens and extract effective representative features. Then, the token features are selected and aggregated to generate sparse representations of WSIs, which are fed into a simple transformer-based classifier (SViT) for slide-level prediction. Experimental results demonstrate that the performance of our proposed SSRViT is significantly improved compared with other state-of-the-art methods in discriminating between adenocarcinoma, pulmonary sclerosing pneumocytoma and normal lung tissue (accuracy of 96.9% and AUC of 99.6%).

Uncertainty-Aware Active Domain Adaptive Salient Object Detection.

Due to the advancement of deep learning, the performance of salient object detection (SOD) has been significantly improved. However, deep learning-based techniques require a sizable amount of pixel-wise annotations. To relieve the burden of data annotation, a variety of deep weakly-supervised and unsupervised SOD methods have been proposed, yet the performance gap between them and fully supervised methods remains significant. In this paper, we propose a novel, cost-efficient salient object detection framework, which can adapt models from synthetic data to real-world data with the help of a limited number of actively selected annotations. Specifically, we first construct a synthetic SOD dataset by copying and pasting foreground objects into pure background images. With the masks of foreground objects taken as the ground-truth saliency maps, this dataset can be used for training the SOD model initially. However, due to the large domain gap between synthetic images and real-world images, the performance of the initially trained model on the real-world images is deficient. To transfer the model from the synthetic dataset to the real-world datasets, we further design an uncertainty-aware active domain adaptive algorithm to generate labels for the real-world target images. The prediction variances against data augmentations are utilized to calculate the superpixel-level uncertainty values. For those superpixels with relatively low uncertainty, we directly generate pseudo labels according to the network predictions. Meanwhile, we select a few superpixels with high uncertainty scores and assign labels to them manually. This labeling strategy is capable of generating high-quality labels without incurring too much annotation cost. Experimental results on six benchmark SOD datasets demonstrate that our method outperforms the existing state-of-the-art weakly-supervised and unsupervised SOD methods and is even comparable to the fully supervised ones. Code will be released at: https://github.com/czh-3/UADA.

PhenoBench: A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domain.

The production of food, feed, fiber, and fuel is a key task of agriculture, which has to cope with many challenges in the upcoming decades, e.g., a higher demand, climate change, lack of workers, and the availability of arable land. Vision systems can support making better and more sustainable field management decisions, but also support the breeding of new crop varieties by allowing temporally dense and reproducible measurements. Recently, agricultural robotics got an increasing interest in the vision and robotics communities since it is a promising avenue for coping with the aforementioned lack of workers and enabling more sustainable production. While large datasets and benchmarks in other domains are readily available and enable significant progress, agricultural datasets and benchmarks are comparably rare. We present an annotated dataset and benchmarks for the semantic interpretation of real agricultural fields. Our dataset recorded with a UAV provides high-quality, pixel-wise annotations of crops and weeds, but also crop leaf instances at the same time. Furthermore, we provide benchmarks for various tasks on a hidden test set comprised of different fields: known fields covered by the training data and a completely unseen field. Our dataset, benchmarks, and code are available at https://www.phenobench.org.

LC-MSM: Language-Conditioned Masked Segmentation Model for unsupervised domain adaptation

Unsupervised domain adaptation (UDA) is an important research topic in semantic segmentation tasks, wherein pixel-wise annotations are often difficult to collect in a test environment due to their high labeling costs. Previous UDA-based studies trained their segmentation networks using labeled synthetic data and unlabeled realistic data as source and target domains, respectively. However, they often fail to distinguish semantically similar classes, such as person vs. rider and road vs. sidewalk, because these classes are prone to confusion in domain-shifted environments. In this paper, we introduce a Language-Conditioned Masked Segmentation Model (LC-MSM), which is a new framework for the joint learning of context relations and domain-agnostic information for domain-adaptive semantic segmentation. Specifically, we reconstruct semantic labels with masked image conditions on the generalized text embeddings of the corresponding semantic class from OpenCLIP, which contains domain-invariant knowledge from large-scale data. To this end, we correlate the generalized text embeddings onto the per-pixel image feature of a masked image that learned the spatial context to further append domain-agnostic language information to the semantic decoder. This facilitates the generalization of our model to the target domain via the learning of context information within individual training instances, while considering cross-domain representations spanning the entire dataset. LC-MSM achieves an unprecedented UDA performance of 71.8 and 62.8 mIoU on GTA→Cityscapes and SYNTHIA→Cityscapes, respectively, which corresponds to an improvement of +3.5 and +1.9 percent points over the baseline method.

A three-stage pavement image crack detection framework with positive sample augmentation

Most pavement crack detection methods based on deep learning rely too much on pixel-wise labels, and are facing with sample imbalance problem. This paper proposes a three-stage pavement crack detection framework with positive sample augmentation by an autoencoder-deep convolutional generative adversarial network (E-DCGAN). Positive samples are the images labeled as crack in the training datasets. Firstly, crack regions are located using graph based on visual saliency (GBVS). Secondly, a convolutional neural network (CNN) model is designed to recognize crack pixels in the located crack regions. The E-DCGAN is explored to generate crack images, which are then added as augmented positive samples in the training dataset of the CNN model. The trained CNN model takes the located crack regions as inputs, outputs the recognized crack pixels. Finally, fine crack detection is realized through region growing. Crack contours are completely depicted pixel-wise based on the recognized crack pixels and the Breadth-first search. Experiments are carried out on two benchmark datasets and in practical applications. The results showed that the framework could detect cracks at the pixel level. Its performance is improved compared with standard fully convolutional network (FCN)-based and U-shaped CNN (U-Net)-based crack detection methods. In addition, the standard FCN-based and U-Net-based methods require accurate pixel-wise annotations, while the proposed framework only requires image-wise annotations. It can be concluded that the proposed framework not only effectively avoids the dependence on pixel-level annotations, improves the sample imbalance issues, but also achieves competitive detection performance over the FCN and U-Net based methods.

The Contrastive Network With Convolution and Self-Attention Mechanisms for Unsupervised Cell Segmentation.

Deep learning for cell instance segmentation is a significant research direction in biomedical image analysis. The traditional supervised learning methods rely on pixel-wise annotation of object images to train the models, which is often accompanied by time-consuming and labor-intensive. Various modified segmentation methods, based on weakly supervised or semi-supervised learning, have been proposed to recognize cell regions by only using rough annotations of cell positions. However, it is still hard to achieve the fully unsupervised in most approaches that the utilization of few annotations for training is still inevitable. In this article, we propose an end-to-end unsupervised model that can segment individual cell regions on hematoxylin and eosin (H&E) stained slides without any annotation. Compared with weakly or semi-supervised methods, the input of our model is in the form of raw data without any identifiers and there is no need to generate pseudo-labelling during training. We demonstrated that the performance of our model is satisfactory and also has a great generalization ability on various validation sets compared with supervised models. The ablation experiment shows that our backbone has superior performance in capturing object edge and context information than pure CNN or transformer under our unsupervised method.